# helm-charts

Collection of some self-made helm-charts. Each top-level directory is a standalone Helm chart that gets packaged and uploaded to a private, classic (index-based) Helm registry by CI.

> **This repo lives on the Gitea instance JLab-Git.** Issues, pull requests, CI and Renovate all run there, and `main` there is the leading copy. github.com is a push mirror with Actions and Renovate **switched off**. The workflow files under `.github/workflows/` are read by Gitea unchanged; "Where this repo lives" below names the places where a Gitea runner behaves differently from a GitHub one, each of them measured rather than assumed.
>
> **The knowledge base is the [wiki](https://git.jk-dev.de/jlab-cluster/helm-charts/wiki).** It holds one page per chart — prerequisites, an installation walkthrough, the complete list of configurable values and the chart's operating traps — plus the long form of the workflow, the pipelines, the style guide and the [lessons](https://git.jk-dev.de/jlab-cluster/helm-charts/wiki/Lessons-Learned) each rule was bought with. This README stays the overview for someone reading the code; anything an *operator* needs is in the wiki. Keeping it current is a required step of the workflow (`CLAUDE.md`), not an afterthought.
>
> **A reading note on `#n`.** Issue numbers in this file point at the **GitHub** tracker. Most of them name *closed* issues, which were not migrated and exist only there. Gitea resolves a bare `#n` against its own counter instead — where, for example, `#22` is an unrelated throwaway pull request rather than the install-test issue this file means. The seventeen issues that were still open at the move are Gitea #5–#21 and #24; the mapping is a comment on Gitea #14.

## Charts

Every chart has a wiki page of its own — `https://git.jk-dev.de/jlab-cluster/helm-charts/wiki/<chart>` — carrying the installation walkthrough, the required 1Password fields and the full values reference. The table below is the one-line version.

| Chart | Deploys | Notes |
|---|---|---|
| `apache-tika` | Apache Tika | Deployment + Service only (no ingress) |
| `bentopdf` | BentoPDF | Stateless Deployment + `config.json` ConfigMap, `subPath`-mounted into the web root and always rendered (empty renders as `{}`) |
| `bulwark` | Bulwark Webmail (JMAP client for Stalwart) | **The companion to `stalwart`, and a pure client**: no mail, no database, no cache, no broker — the messages belong to the Stalwart release named by `bulwark.jmapServerUrl`. StatefulSet with **one** PVC at `/app/data`, carrying all four of Bulwark's directories (admin config, admin state with the audit log, encrypted user settings, telemetry) as subdirectories — one volume because snapshots are atomic per volume and never across them (the `garage` lesson, #81). No ConfigMap, so no `checksum/config` (§9 has nothing to attach to); the `op://` paths are covered by `checksum/secret-ref`. **The defining property, and it governs every value: the VOLUME BEATS THE CHART.** `configManager.get` resolves `admin override > env var > default`, the admin dashboard may write *any* key, and nothing says it just disconnected the chart — so a `helm upgrade` with a changed value can succeed and change nothing. Upstream documents this only for the server URL; it is true of everything. Exactly three values escape it (`SESSION_SECRET(_FILE)`, `ADMIN_PASSWORD`, `BULWARK_TELEMETRY`), and `bulwark.adminConfigReadOnly` closes it at a price that is **threefold and measured**: it freezes the settings policy (and with it the only way to move Web Push off the vendor relay), it freezes an override that *already* exists rather than removing it, and on a **fresh** install it stops `admin.json` from being written at all — so the admin dashboard never comes up and answers HTTP 500 while the pod stays Ready. Hence the default `false`. Sign-in is **OIDC-only** (`oauth.only`), and the way back out of a lockout is **not** `helm upgrade` with it `false` — `oauthOnly` is read through the same precedence chain, so the repair is removing the override in the **admin dashboard**, which is why `ADMIN_PASSWORD` is set at all (it generates no random password and logs nothing — the `#65` analogy does not hold here). Its **redirect URI carries the UI language** (`/<locale>/auth/callback`, 27 locales, and the language is not in the visible URL), so a single registered URI is the likeliest cause of a failed first sign-in. Both `SESSION_SECRET_FILE` and `OAUTH_CLIENT_SECRET_FILE` exist and are used — the better path `garage` and `keycloak` did not have — mounted **without `subPath`** so that a rotation takes effect without a restart, since both are re-read on every use. Two backup statements, both sharp: without the session secret the synchronized settings are unreadable **out of an intact volume**, and if the dashboard's OAuth setup was ever used the client secret sits in `config.json` on that volume in **cleartext**. All three probes are a plain GET on `/api/health` — **not** HEAD, which a `HTTPGetAction` cannot send — and the limit is written at the value: it carries process, HTTP acceptance, a free event loop and Next.js routing, and **nothing** about Stalwart, so with the mail server gone the pod is green. Served only at a host **root** (`NEXT_PUBLIC_BASE_PATH` is compiled in), one replica (single RWO PVC), two emptyDirs for `readOnlyRootFilesystem` (`/tmp`, `/app/.next/cache` — image optimization is active because `images.unoptimized` is in the lite branch only) |
| `calibre-web-automated` | Calibre-Web Automated | StatefulSet, extra consume-PVC |
| `cyberchef` | CyberChef | Stateless Deployment. **The Gateway-API pilot (#114)**: the first chart offering `httpRoute.enabled` as the alternative to `ingress.enabled` — `HTTPRoute` with `parentRefs` only, no Gateway and no certificate, the two modes mutually exclusive. It is also the only chart whose install-test runs in Gateway mode, because that test installs one configuration per chart |
| `freesailarr` | gluetun VPN media stack (qbittorrent, prowlarr, flaresolverr, radarr, recyclarr, sonarr, seerr, bazarr, and mediathekarr off by default) | Single-pod StatefulSet: all apps share gluetun's network namespace; config volumes via volumeClaimTemplates, media PVCs as existingClaims (optional per-category `subPath`, so one claim can serve all three categories). One ClusterIP Service per app (mediathekarr's carries both of its ports); no global `ingress.domain`. Ingress in two shapes that mix: one host per app (default, each app's host required), or one shared `ingress.host` with the apps as sub-paths (documented ingress exception below) |
| `garage` | Garage (S3-compatible object store) | **One** node, `replication_factor = 1` — a deliberate trade with its price named (Gitea #81): the volume layer underneath already replicates and encrypts, so Garage replication on top would be nine times the raw data across two layers that know nothing about each other. What that costs is that the factor applies to the **metadata** too, where a corrupted database is upstream-documented as "generally not recoverable" and a replicated volume replicates the corruption. Hence three settings that are mandatory rather than optional: `metadata_fsync = true` (**off** in Garage's defaults), the automatic metadata snapshots, and those snapshots on a **second PVC** — a file-system snapshot of a live metadata database is not a restore point, so the volume backup covers the blocks and not the index. The first PVC carries `metadata_dir` **and** `data_dir` as two subdirectories, because volume snapshots are atomic per volume: that is what keeps index and blocks consistent with each other. Two further properties shape the whole chart: the image is **`FROM scratch`** and contains exactly one file, `/garage` — no shell, so **exec probes are impossible here** and an init container would need a second image — and the RPC secret has to arrive as an **environment variable**, because `rpc_secret_file` rejects any file with a group bit set while the kubelet sets one (the reasoning sits at the value). Started with `garage server --single-node`, which creates the cluster layout in-process before any listener opens and is idempotent on the layout version, and with `--default-access-key`/`--default-bucket`, which **import** key and bucket from the 1Password item — so the vault is the source of truth instead of a copy of what the server generated. Readiness is the admin API's `/health`, liveness and startup deliberately are **not**: `/health` answers 503 until a layout exists, so a node waiting for one would be restarted forever |
| `gotenberg` | Gotenberg | Deployment + Service only (no ingress) |
| `homeassistant` | Home Assistant | StatefulSet; the only chart carrying Helm tests (`templates/tests/`) |
| `keycloak` | Keycloak (OpenID Connect / SAML identity provider) | **Two** replicas with `cache=ispn`, clustered through Keycloak's default `jdbc-ping` stack — discovery runs over the PostgreSQL that is there anyway, so no headless Service and no JGroups wiring. Deployment, **no PVC** (Keycloak 26 keeps everything in the database, user sessions included). **The chart has two init containers and needs both** (Gitea #83): `db`, `health-enabled` and `metrics-enabled` are Keycloak *build* options, the published image is built with their defaults, and a runtime change makes Keycloak rewrite `/opt/keycloak/lib/quarkus` at start — which `readOnlyRootFilesystem: true` forbids and which upstream describes as a start that "creates an image that is mutable". An empty `emptyDir` over that path does not work either, because `QuarkusEntryPoint` loads `quarkus/quarkus-application.dat` before it dispatches any command, `build` included. So one init container **seeds** the directory into a volume, the next runs `kc.sh build` into it, and the serving container runs `start --optimized` — every container keeping the full hardening baseline. Build options therefore exist in exactly **one** place and a build option in `keycloak.env` fails the render. First chart with a `PodDisruptionBudget` (§11) and first `quay.io` image (hence the entry in `ci/trivy/data/ksv0125.yaml`). Two Services (8080, and the management port 9000 **never** published — it serves `/health/*` and optionally `/metrics` unauthenticated) and two Ingresses — plus one routing object per `keycloak.realmHosts` entry (Gitea #122), empty by default, which gives a realm a hostname of its own by sending that hostname's bare `/` to `/realms/<realm>/account/`: `app-root` in Ingress mode, an `Exact` match with a `RequestRedirect` filter in Gateway mode. Those entries carry a **required** per-entry TLS secret name and an **optional** per-entry issuer, the documented deviation in §4, because the hostnames they publish already have certificates and often share one. **The admin hostname is not an access control**: `hostname-admin` only decides which URLs the server generates, and upstream states that it "does not prevent accessing the Administration REST API endpoints via the frontend URL" — restricting that API is reverse-proxy work, and the chart renders no `/admin` deny rule. `ingress.domain` is read as `KC_HOSTNAME` (as a full URL, which upstream requires alongside `hostname-admin`), so it stays required with the Ingress off. Liveness is `/health/live` and deliberately not `/health` or `/health/ready` — those include the database, and a liveness probe on them restarts the whole cluster for as long as the database is away. **`keycloak.metrics.enabled` defaults to `true` for the readiness probe, not for the metrics**: upstream's table of health checks carries a "Requires Metrics" column and the *database* check is the row that says yes, so with it off `/health/ready` answers 200 while the database is gone — the `urlaubsverwaltung` shape (#218) with the dependency hidden behind an option whose name says observability. Two traps at the values: upstream's `db-pool-max-size` default of 100 would ask a default PostgreSQL for 200 connections across two replicas (the chart ships 20), and two releases sharing one database **merge into one Infinispan cluster** unless the cluster names differ |
| `matrix-synapse` | Matrix Synapse | StatefulSet plus an nginx delegation Deployment/Service |
| `minecraft-router` | itzg/mc-router | Deployment + routing ConfigMap + NodePort Service (TCP only, no ingress); `mappings` is a required value |
| `minecraft-server` | itzg/minecraft-server | StatefulSet + PVC + Service + init-script ConfigMap (no ingress); service type is a value (`ClusterIP` by default, routed via `minecraft-router`). **Only the newest Java line is maintained** (chart 5.0.0, #158): the directory carries one line, older ones stay published as frozen artifacts (java8/java17 at `4.0.0`, java21 at `4.1.0`) and receive no further chart changes — publishing once per line out of one directory is what made the lines drift. To run an older JVM with the current chart, set `image.tagOverride`; the chart then checks that `minecraft.version` fits the Java line of the **effective** tag and fails the render with both values named. Two cases pass unchecked on purpose: a tag without a `java<N>` part and a `minecraft.version` that is not a plain numeric `x[.y[.z]]`. `image.tagOverride` is deliberately outside Renovate's reach — `renovate.json5` tracks `appVersion` only. Since chart 2.0.0 the tag is an **immutable dated upstream release** (`appVersion: <release>-java8` / `-java17` / `-java21` / `-java25`, e.g. `2026.8.2-java8`) instead of the continuously rebuilt rolling line tag. Pilot for the strict `values.schema.json` (#68), which is now binding for every chart |
| `outline` | Outline | StatefulSet, separate DB/Redis secrets |
| `paperless-ngx` | Paperless-ngx | Largest chart: consume CronJob, SMB-connector PV/PVC, pre/post-consumption scripts ConfigMap. Optional `valkey` subchart — **this repo's own chart, from the published index**, off by default and gated on `valkeySubchart.enabled` (not `valkey.enabled`, which the subchart's strict schema rejects). Configured as a **queue** (PVC + `noeviction`), unlike `stalwart`'s cache: `PAPERLESS_REDIS` is the Celery broker, and `allkeys-lru` evicts across the whole keyspace, where a backlog of waiting consume tasks is exactly what LRU reads as cold. The password comes from the **same 1Password item** as the Redis connection, in a third field `auth-conf` holding the complete line `requirepass <password>` — so the password sits in that item **twice** and a rotation has to touch both fields. Two more traps: with the subchart on, `redis-connection-url` in the vault **must** be repointed at `<release>-valkey-service:6379` or the bundled Valkey runs unused, and `valkeySubchart:` must not be erased — a null override deletes the key, Helm reads an unresolvable `condition:` as *enabled*, so erasing the switch turns the subchart **on** (rejected by `values.schema.json`). Never change `valkey/` and this chart in the same PR, see "Chart dependencies" |
| `patchmon` | PatchMon | One Deployment (the 2.x server serves frontend and `/api/*` from one process) plus an optional guacd sidecar; OIDC-only sign-in |
| `rallly` | Rallly | Deployment, DB/OIDC/SMTP secrets |
| `rss-bridge` | RSS-Bridge | Deployment + whitelist ConfigMap; `ingress.hosts` is a required list of host + optional `tlsSecretName`, so several hosts can keep their own existing certificates (documented deviation from the one-TLS-secret rule, see its `values.yaml`) |
| `samba` | Samba | StatefulSet, users secret + config ConfigMap |
| `satisfactory` | Satisfactory game server | StatefulSet, no ingress |
| `searxng` | SearXNG | Stateless Deployment + settings ConfigMap (`use_default_settings: true` plus a free-form `settings.extra` passthrough); the signing key is a required `OnePasswordItem`, an external Valkey an optional second one. `ingress.domain` is read twice — Ingress host and `SEARXNG_BASE_URL` — so it stays required with the Ingress switched off |
| `stalwart` | Stalwart mail server (SMTP, IMAP, POP3, JMAP, ManageSieve) | A webmail client for it is the separate `bulwark` chart (above), pointed at this release's JMAP endpoint — the two are only useful together, and nothing in either reports the other being away. **The chart configures almost nothing, and that is the chart's defining property.** Since v0.16 Stalwart keeps its whole configuration — every listener, the blob store, the in-memory store, TLS objects, domains and the cluster roles — inside the data store and manages it over JMAP; the only file on disk is a `config.json` that names the data store and nothing else. So the chart renders that ConfigMap (secret-free: `authSecret` uses the `EnvironmentVariable` variant, so the password arrives by `secretKeyRef` and `checksum/config` stays meaningful), one StatefulSet per entry of the `nodes` **map** with `STALWART_ROLE` and `STALWART_HOSTNAME`, one NodePort Service **per port** (not per group — a NodePort is cluster-unique, so `full` and `mx` can only take SMTP behind the same Service, selected by a per-port pod label), a per-group ClusterIP `http` Service for the Ingress, per-group headless Services and NetworkPolicies, and a cert-manager `Certificate` for the mail ports. Everything else is operator work in the WebUI, which is why `NOTES.txt` carries an eight-step first install. Four traps worth knowing before reading the chart: **the role lookup fails open** — a node whose `ClusterRole` is missing from the database enables every listener *and* every task while staying Running with a green probe, so `mx` ships at `replicas: 0` and the NetworkPolicy is a ceiling independent of what was bound; **`NET_BIND_SERVICE` is a documented deviation needed to `exec`, not to bind** (the image sets `setcap cap_net_bind_service=+ep`, so `drop: [ALL]` alone makes the kernel fail `execve` — removing it as "unused" stops the chart starting); **`externalTrafficPolicy` must be `Cluster`** (no pod runs on a control-plane node while the HA proxy delivers to them, so `Local` would drop every packet) and the source address it costs is restored only by the HA proxy's PROXY v2 header; and **`/healthz` is rate-limited to 100 req/60s per source IP**, which is a node IP for kubelet probes, so the probe intervals are calculated rather than chosen. A first install needs `bootstrap.enabled: true` because the setup wizard appears only when `config.json` is *absent*. Optional `valkey` subchart — **this repo's own chart, from the published index**, off by default and gated on `valkeySubchart.enabled` (not `valkey.enabled`, which the subchart's strict schema rejects); never change `valkey/` and this chart in the same PR, see "Chart dependencies" |
| `sure` | Sure (we-promise, the maintained Maybe-finance fork) | One StatefulSet with **two containers**: the Puma web process and a Sidekiq worker, sharing one ReadWriteOnce PVC at `/rails/storage` — that shared volume is the reason they live in one pod. Only the web container migrates (the image entrypoint runs `db:prepare` solely for the `./bin/rails server` command, so the worker is started through `args:` and skips it). Three 1Password items (app/database/redis); Postgres and Redis stay external. Readiness deliberately probes `/` rather than Rails' `/up`, which is a boot check and touches no database; the worker's three probes ask `sidekiqmon` for this pod's own heartbeat. `ingress.domain` is read for `APP_DOMAIN` and the WebAuthn relying party, so it stays required with the Ingress switched off |
| `teamspeak-server` | TeamSpeak server | StatefulSet, no ingress |
| `template-chart` | — | Scaffold for new charts (placeholder `xxx`), never published |
| `urlaubsverwaltung` | Urlaubsverwaltung | Deployment, DB + OIDC secrets |
| `valkey` | Valkey | One instance per application (cache or queue backend), replacing the shared `cluster-redis` whose single ACL file coupled six applications together. StatefulSet with one replica + ClusterIP Service, no cluster mode, no Ingress and no value to add one. The password arrives as a **finished configuration line** (`requirepass <secret>`) in the 1Password field `auth-conf`, is mounted as a file and `include`d by the config — never in argv, never in the environment; an init container refuses to start the pod unless that line is really there, because the alternative failure is an open server that looks healthy. `maxmemory` is **derived from the computed memory limit** and a configured value above it aborts the render. Persistence and eviction are a pair, not two switches (queue = PVC + `noeviction`, the default; cache = emptyDir + `allkeys-lru`). First chart carrying a **NetworkPolicy** (on by default, Gitea #5), and the first whose probes had to be proven capable of failing: `valkey-cli` exits 0 on an error reply unless `-e` is given |
| `voucher-vault` | VoucherVault | StatefulSet, DB + OIDC secrets |
| `ytdlp-web-player` | ytdlp_web_player | Stateless Deployment, video cache and tool cache as emptyDirs (no PVC). **The first chart with a dependency** (`stalwart` and `paperless-ngx` followed): an optional `oauth2-proxy` subchart, off by default, wired in front of the app because the application ships no authentication at all — see "Chart dependencies" below |
| `zipline` | Zipline | StatefulSet, DB secret |

## Versioning

`Chart.yaml` uses the scheme `version: <chart-semver>+up<app-version>` (e.g. `1.4.0+up1.9.2`) — see the [chart style guidelines](#chart-style-guidelines) for the full rules (suffix normalization, semver bump levels).

- Bump the **chart version** in every change that touches a chart directory — a push to `main` publishes the chart.
- **App-version updates go through PRs** (so the validate and install-test pipelines run). The manually dispatched `Change App-Version` workflow (rewrites `version:`/`appVersion:` and commits directly to `main`) exists but bypasses both pipelines. For app **major** jumps, update stepwise: first to the last release of the old major line (own PR, merge, publish), then to the newest major.

## CI/CD

- `publish-<chart>.yml` — one thin workflow per chart; triggers on pushes to `main` touching `<chart>/**` (or manually) and calls the reusable workflow below. **A push to a chart directory on `main` publishes that chart**, so the chart version must be bumped in the same change.
- `publish-helm-chart.yml` — **not in this repository any more.** The reusable workflow lives in [`workflow-util/pipelines`](https://git.jk-dev.de/workflow-util/pipelines), the shared building-block repo, and the callers here bind it to a tag: `uses: workflow-util/pipelines/.gitea/workflows/publish-helm-chart.yml@v3` (Gitea #66). It still does the same thing — `helm package`, then a single HTTP `POST` of the packaged `.tgz` to the private chart registry, which is a **classic, index-based** Helm registry and not an OCI one, so `helm push` is not used, there is no `helm registry login` session, and every request carries its own Basic auth. What changed is who supplies what:
  - **`registry-owner: ${{ github.repository_owner }}`** in each caller's `with:`. The owner used to be derived inside the workflow (#63); it is now a required input with no default, so the expression is evaluated **here**, in the calling repository, where the value is measured. That is `pipelines`' house rule — a shared block never guesses an owner segment — and it is why every caller carries that one extra line.
  - **`secrets: inherit`**, unchanged, for `OP_CONNECT_TOKEN`.
  - the repository **variable `OP_CONNECT_HOST`**, which replaced the Connect host that used to be hardcoded in our copy. A shared block must not carry one tenant's infrastructure address; the block's first step fails and names the variable when it is unset.

  The registry credentials and the target URL still come from 1Password Connect (item `op://GitHub-Actions/jlab-registry-access/` with `username`, `password`, `charts-url`). `charts-url` holds only the **base up to `/api/packages`** and the block composes `<charts-url>/<owner>/helm/api/charts`, so **the publish target and its credentials appear nowhere in this repository.** `ci/helm-dependencies.sh` is still run before `helm package`, and it is still **our** script: the block's checkout clones the calling repository, so it runs from this tree.

  **What this does *not* claim any more**: that no registry address is in the repo at all. That wording was true until a chart took one of our own published charts as a subchart — `stalwart`'s `Chart.yaml` names `https://git.jk-dev.de/api/packages/jlab-cluster/helm` as a dependency `repository:`, because a Helm dependency is resolved from a URL in the chart and there is nowhere else to put it. The narrower claim above is the one that matters and it is unchanged: that is the **read** path, the index is anonymously readable (measured), so it carries **no credential** and grants nothing a `git clone` of this repo does not already imply. The host itself has always been in `CLAUDE.md`; what is new is only the read-only path segment.

  The reasons behind the upload step are **not repeated here**, because they are no longer ours to keep current: why both halves of the composed URL are guarded, why the credentials go in through `curl --config -` instead of `--user`, why `--fail-with-body` is what makes a failed upload fail the job, and why the file name is built rather than globbed, are all on the [`publish-helm-chart` wiki page](https://git.jk-dev.de/workflow-util/pipelines/wiki/publish-helm-chart) of that repository. Nothing was dropped in the move — the upload step's shell body went across unchanged except for two error messages that name the owner as an input.

  Note: the registry **accepts a re-upload of an already published version** (HTTP 201) and replaces the stored artifact, so a republish of an existing version is still silent — see #160. That is also what makes a manual `workflow_dispatch` of any `publish-<chart>.yml` a harmless way to prove the shared block works.
- `change-app-version.yml` — manually dispatched app-version bump; the chart must be listed in its `chart-name` choice list.
- `ci/helm-dependencies.sh <chart-dir>` — not a workflow but the step they all share: fetches a chart's declared dependencies into `charts/`, and returns silently for the charts that declare none (all but three: `ytdlp-web-player`, `stalwart`, `paperless-ngx`). It runs in **seven** places, because every helm command this repo builds on exits 1 while a declared dependency is missing: before lint and package in `validate-charts.yml`, before package in the shared publish block (which is why that block names this script as a prerequisite of its callers — see above), and inside `ci/run-install-test.sh`, `ci/run-quantity-render-test.sh`, `ci/run-selector-match-test.sh`, `ci/run-httproute-match-test.sh` and `ci/run-trivy-config.sh`. See "Chart dependencies" in the style guidelines for what that costs.
- `validate-charts.yml` — PR pipeline: detects which charts a PR touches and, for exactly those, runs `helm lint --strict`, `helm package`, and a chart-version bump check (`template-chart` is exempt from the version check). PRs without chart changes skip the build matrix. After the build stage, an **install-test** job (the same changed-charts matrix, minus the charts listed in `ci/excluded-charts.txt`) installs each changed chart into an ephemeral [k3d](https://k3d.io) cluster on the runner and requires every workload to become Ready — see below. Both matrices follow the changed chart directories, so a PR that only changes the harness under `ci/` exercises nothing by itself; the Trivy scan has a fixed-sample fallback for that case, the install-test has none — bump the affected chart's patch version to pull it into the matrix, or run the all-charts install-test after the merge. The **resource-quantity render test**, the **selector-match render test** and the **HTTPRoute render test** are the three exceptions to that rule: all of them run over every chart on every PR — see below.
- `install-test.yml` — reusable workflow (`workflow_call`) holding the install-test **job** itself: the runner-label expression, the pinned `kubectl` with its skew check, `envsubst`, the containerised-job detection, the per-leg k3d cluster and the `freesailarr` VPN credentials. Its one input is a JSON array of chart names; its callers decide *which* charts are tested and whether the job runs at all, never *how*. Two callers: `validate-charts.yml` and `install-test-all-charts.yml`. **It is not copied for the second caller** — a second copy is a second chance to test differently, unnoticed, the same argument that gives `ci/excluded-charts.txt` exactly one parser.

  **It stays in this repository, and that is a decision, not an oversight** (Gitea #66, decision 4). It is reusable in form but not in substance: it is welded to the `ci/` fixtures, to the `ci/excluded-charts.txt` parser and to this repository's chart matrix. A building block that only one repository can use is not shared, it is merely moved somewhere else — and moving it would buy nothing while putting a tag between us and a file we change often. `publish-helm-chart.yml` was the opposite case: it takes a chart name and a registry owner and knows nothing about this repository, which is exactly why it could go. This note exists so the question is not asked again in half a year.
- `install-test-all-charts.yml` — manually dispatched **all-charts install-test**: the same job, with a matrix over every installable chart instead of the changed ones. Optional input for a subset of charts (empty = all). See below for when to run it.
- `sonar.yml` — **SonarQube analysis**, a hull like the publish ones: it triggers on pushes to `main` (no `paths` filter — the analysis looks at the whole tree) or manually, and calls `workflow-util/pipelines/.gitea/workflows/sonar-analysis.yml@v4` with `secrets: inherit`. The project-specific half is `sonar-project.properties` in the repo root. **It is not a gate** and it does not block anything — see below.

### Where this repo lives, and what a Gitea runner does differently

Everything below was measured on this instance, not taken from documentation, and it is the reason for every deviation in `validate-charts.yml`. How the move got here is not repeated — that is Gitea #14.

**Reusable workflows with `secrets: inherit` work**, so the 25 thin `publish-<chart>.yml` hulls run unchanged — including now that the workflow they call lives in another repository (Gitea #66). One quirk worth knowing before debugging the wrong thing: a *missing* secret does not fail the job, it fails job **creation**, so the run shows no steps at all.

**Runner labels mean different machines than on GitHub.** `ubuntu-latest` here has **no Docker socket**; `ubuntu-docker` has one. Exactly one job cares — the install-test, because k3d is a Docker client — and its `runs-on` is therefore an expression rather than a fixed label, so the same file still works on a GitHub runner where `ubuntu-docker` does not exist. Every other job (lint/package, the quantity render, the Trivy scan) needs no socket and stays on `ubuntu-latest`.

**A matrix runs concurrently inside ONE job, under ONE status context.** This is the single most consequential difference and it has two halves. Operationally: all legs share one Docker daemon, which is what makes a per-leg resource name mandatory (see the cluster name below). For branch protection: **a required check per chart is not possible**, because there is only one context for the whole matrix — and its name carries the *unexpanded* matrix expression, e.g. `Validate Helm Charts / Install-test ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }} (pull_request)`. Required-check names have to be read off a real run; they cannot be copied from GitHub. That example name is from before the install-test job moved into the reusable `install-test.yml` (Gitea #52) — the matrix expression now reads `${{ fromJSON(inputs.charts) }}` and the context may carry the calling job's name as well, so **read the current name off the next chart PR before using it anywhere**. What *is* required on `main` today is **`validate-charts`**, as the single context it reports under, alongside a **review**: per chart is the part that is impossible, not required checks as such. Both are what makes Renovate's `platformAutomerge: true` safe — see "PR policy".

**Expected runtimes**, so that a later deviation is visible as one:

| | measured here | GitHub, same commit |
|---|---|---|
| `Create k3d cluster` | 19–25 s | 22–23 s |
| `install-test` job, two charts in the matrix (cyberchef + freesailarr) | 210–216 s over three runs, spread 6 s | ~145 s wall, two parallel VMs |
| `install-test` job, freesailarr alone, ten containers and a real tunnel | 225 s (measured on the MediathekArr change, Gitea #20) | not comparable — Actions there were already off |
| `install-test` job, **20 charts** in the matrix — the accidental all-charts run of Gitea PR #48 | **12m41s**, one job, all legs concurrently on one Docker daemon; 19 of 20 charts Ready, the twentieth is Gitea #51 | never ran there |

The runner is **not slower** — cluster creation is a touch faster. The difference in total time is **structural, not performance**: two matrix legs share one runner here, while GitHub gives each leg its own VM. Reading it as "Gitea is slower" leads to the wrong conclusion.

**`aquasecurity/setup-trivy` cannot be used.** It runs `actions/checkout` against `aquasecurity/trivy` on github.com and hands it `github.token` — which here is a *Gitea* token. github.com answers 401 and checkout ends on `fatal: could not read Username for 'https://github.com'`, naming neither the action nor the token. Its `actions/cache` step is inert here as well. The workflow downloads the release tarball instead: one request, same behaviour on either forge.

**Tools are installed, not inherited — and the two cases differ on purpose.** `kubectl` is **pinned** to match `K3S_IMAGE` and guarded against version skew, because a client's distance to the API server changes what the test *means*, and a version that arrives with the runner image drifts with every image rebuild. `envsubst` is only **checked for and fetched if absent**, because it substitutes text and the release doing so does not change the result; pinning it would be upkeep without value. That GitHub's image happens to ship both does not make either unnecessary — it only makes the drift invisible there.

**Containerisation is detected, not inferred from the platform.** k3d publishes the API server port on the Docker *host*; when the job is itself a container — as every Gitea Actions job is — that host is the dind container, so the `https://0.0.0.0:<port>` k3d writes into the kubeconfig points at the job container, where nothing listens. The workflow asks the daemon about its own container instead of asking which forge it is on: a self-hosted **GitHub** runner in a container has the identical problem, and a condition that says "GitHub" while meaning "not containerised" is the kind of lie this repo exists to remove.

Still unmeasured, because they never stood in the way: `/dev/fuse` and user namespaces. Said as unmeasured rather than as fine.

#### What a GitHub runner had been providing quietly

Getting the install-test to run here surfaced six assumptions that had never been written down anywhere. Five of them say something about the **runner**, one about **our code**, and the split is the point.

The five environment ones are the harmless kind: they fail loudly, immediately, and are fixed on the node side (`cluster-config`), not here — Docker bridges built at MTU 1500 against a pod network of 1450 (small responses fit in one packet, a 17 MB tarball does not, which is why it looked like a blocked host); `fs.inotify` limits too low for k3s, so containerd never started and the node never registered; no `kubectl` and no `envsubst` in the image; and the kubeconfig pointing at `0.0.0.0`.

The sixth is different, and it is the one to remember.

> **The k3d cluster name was hard-coded as `install-test` — and that was wrong on GitHub too.** It simply could never show there: each matrix leg gets its own VM with its own Docker daemon, so two legs never saw each other's cluster. Here, where all legs share one daemon, it collides immediately — one leg logs `Re-using existing network`, the other dies on `The container name /k3d-install-test-server-0 is already in use`, and the first leg's rollback then deletes the shared network out from under the second. Two red legs, and neither message names the cause.
>
> For two years the roomier environment carried that bug instead of showing it. **A generous environment is no friend when you want to know whether something is correct.**

#### Two rules for every check added here

Both were learned by getting them wrong in this repo, and both apply to any new verification step, not just to clusters.

1. **A check that a broken system passes is worse than no check.** `kubectl get nodes` exits **0 on an empty list**, so a readiness check written that way went green against a cluster whose node had never registered — and the failure then surfaced five minutes later, somewhere unrelated. `kubectl wait --for=condition=Ready nodes --all` is not the shorter way to write it: with zero nodes the condition is vacuously satisfied. Assert the thing you mean (*at least one node, and Ready*), not that the command answered.
2. **Green does not mean checked.** A passing job earns exactly as much trust as the assertions inside it. The open case of this kind is Gitea #24: the images in `ci/<chart>/fixtures.yaml` are not followed by Renovate, so the install-test can keep passing against fixture infrastructure that is years stale. **Not solved** — named here so it is not mistaken for solved.

### Resource-quantity render test (all charts, every PR)

`ci/run-quantity-render-test.sh [chart …]` (no argument = all charts) renders every chart twice with the fixtures the install-test and the Trivy scan already use, injecting a bare integer memory request at **every** `resources` block the chart's `values.schema.json` declares:

| Case | Injected | Expected limit | Catches |
|---|---|---|---|
| `bare-int` | `requests.memory: 1048576`, `limitFactor: 2` | `2097152` | render aborts |
| `fraction` | `requests.memory: 999999`, `limitFactor: 1.5` | `1499998.5` | quantity in exponential notation |

The two cases exist because the bug behind #157 — quantities formatted with `printf "%v"`, where the `float64` the null-safe fallback helpers produce flips to exponential notation from about `1e6` — has a loud and a silent half. Loud: the reformatted quantity carries the suffix `e+06`, the helper rejects it, `helm template` exits non-zero. Silent: the *product* of a fractional `limitFactor` and a bare number is reformatted **after** the suffix check, so the render succeeds and the manifest carries `memory: "1.4999985e+06"` — rejected by nothing until the API server sees it at apply time. A check on the exit code alone therefore proves only half the case; the second case searches the **rendered text** for a resource quantity in exponential notation. `999999` stays below `1e6` on purpose, so the fraction case isolates the second formatting site instead of tripping over the first.

Why all charts rather than the changed ones: the resources helper is copied into every chart's `templates/_helpers.tpl` rather than shared through a library chart. The regression arrives by copy — a new chart started from an older chart, or from `template-chart` before the fix — and in that PR the copied bug is in no diff anyone reads. Running over everything also removes the list nobody would maintain: adding a chart enrolls it automatically. Two renders per chart, no cluster, a few seconds for the whole repo; the job depends on nothing else and finishes well inside the `build-chart` runtime.

Two details worth knowing before touching it:

- **The resources paths come from `values.schema.json`, not `values.yaml`** — it is JSON, so `python3` reads it without pyyaml (same runner constraint as `ci/run-trivy-config.sh`), and every chart-owned key must appear there anyway. A block counts when it is `#/definitions/resources` or carries `limitFactor`/`requests` itself, which is what keeps `minecraft-server`'s JVM-sizing `minecraft.resources` out. A chart whose templates render a resources block while its schema declares none **fails** the test with exactly that message: the schema is then out of sync with the chart.
- **A chart that never reaches the helper is reported, not counted as passing.** `gotenberg` sets every computable limit explicitly in its fallback defaults, and since a `null` never deletes a default there, no override can free them — nothing is multiplied, so nothing is proven. Those charts appear under "helper not reached" in the job summary rather than as a green tick for a code path that was never entered.

### Selector-match render test (all charts, every PR)

`ci/run-selector-match-test.sh [chart …]` (no argument = all charts) renders every chart with the same fixtures the install-test uses and checks that **every label selector the render contains really matches the labels of a pod template in that same render**. Two kinds of object are covered, and both of them fail *silently* when their selector matches nothing:

| Object | What a non-matching selector does | Who notices today |
|---|---|---|
| `topologySpreadConstraints` | the constraint is a condition over **zero** pods and is therefore trivially satisfied: nothing is distributed, no event, no log line | nobody — and with `whenUnsatisfiable: DoNotSchedule` the object reads like a guarantee |
| `PodDisruptionBudget` | the budget does its arithmetic over zero pods and never blocks an eviction, so `kubectl drain` takes every replica at once | nobody — `kubectl get pdb` shows an object that looks like protection |

Neither is a render error, neither fails the install-test, and neither shows up in the diff of the PR that causes it: the chart that breaks it is usually a chart that **renamed a label somewhere else**. Charts answer the first half of that by keeping the selector and the pod labels in one helper (`stalwart.groupLabels`, `keycloak`'s `app` label) — but one definition only helps while nobody writes a second one, and that is the half a test can hold. Introduced with Gitea #96, where the chart-owned spread constraint made the trivially-satisfied case reachable by design rather than by accident.

Three things to know before touching it:

- **Helm is the YAML parser, through the one-template chart `ci/yaml2json`.** The runners have neither `pyyaml` nor `yq` (measured; the same constraint `ci/run-trivy-config.sh` and `ci/run-quantity-render-test.sh` work under), and a YAML reader written in `awk` was the obvious alternative and the wrong one: its failure mode is a parser that quietly reads nothing, finds no selector, reports no mismatch and leaves the test green for a chart it never looked at. So the rendered manifest stream is handed to that chart as a **values file** and comes back as JSON, which `python3` reads from the standard library. The one textual step is the document split on a line that is exactly `---` in column one, and getting *that* wrong does not produce a false pass — the fragment stops being valid YAML and `helm` exits 1.
- **What is deliberately not checked, named so the gap is not mistaken for coverage.** `podAffinity`/`podAntiAffinity` are out of scope: their semantics are relative to pods that exist, so a term matching nothing is a legitimate configuration rather than a mistake. And a selector expressed **only** through `matchExpressions` is **reported as "not checked", never counted as passing** — evaluating set-based requirements means reimplementing label-selector semantics, which is a second parser by another name.
- **Two anti-vacuum guards, because this test can be green over nothing.** Both objects are rendered only above a replica threshold, and most charts' CI fixtures sit below it. `ci/selector-match/<chart>.yaml` is a render-only overlay applied **last**, whose only job is to push a chart over that threshold — and a chart that *has* one and still renders no selector **fails**, since that overlay existing is the whole claim that something was checked. On top of that the run fails if **no selector at all** was checked anywhere. Today `stalwart` is exercised through its overlay (two replicas in both node groups plus an `externalServices.valkey.host`, without which the chart's own "cluster with no in-memory store" abort fires) and `keycloak` through its ordinary install-test fixture, which already runs two replicas.

Measured: 30 charts, **5 selectors verified in ~14 s**, no cluster. The fail path was exercised rather than assumed — renaming `node-group` to `nodegroup` in a hand-written copy of the PDB selector produced two `FAIL` lines naming both the selector and the labels actually present, and a non-zero exit; the same was done for the spread constraint's selector.

### HTTPRoute render test (all charts, every PR)

`ci/run-httproute-match-test.sh [chart …]` (no argument = all charts) renders every chart with the same fixtures the install-test uses and checks every `HTTPRoute` in the result against the `Service` objects of that **same** render. Introduced with the Gateway-API pilot (Gitea #114); three failure modes, all silent:

| Finding | What it does | Who notices today |
|---|---|---|
| `parentRefs` empty | the route attaches to **no** Gateway: created, schema-valid, rejected by nobody, and it routes nothing | nobody — no event, no condition, the only symptom is a hostname that does not answer |
| `backendRefs` naming a Service or a port that does not exist | the reference is resolved **by name** at runtime and validated by nothing | nobody in CI — `ResolvedRefs: False` lands in a route status that only a Gateway controller writes, and none runs here |
| a prefix stripped with `ReplacePrefixMatch` while the same route carries no `Exact` match on that prefix with a `RequestRedirect` filter (Gitea #118) | the **slashless** form of the path is rewritten to `/` and answered **200** by the app, under a document URL that keeps the prefix off — so every relative link in that page resolves against the host root, which on a shared host is a *different* application | nobody — the app's own assets and XHRs are served by its neighbour, and the symptom is a half-working UI, not an error |

**The real CRDs in the install-test close none of these gaps**, which is the whole reason this job exists next to them: the Gateway-API schema happily accepts an empty `parentRefs` list, a `backendRefs` entry pointing at a Service that was never created, and a prefix-stripping rewrite with no redirect beside it. The CRDs prove the object is *well-formed*; this test proves it *points somewhere*, and *somewhere right*. The port half is a genuine new exposure rather than a hypothetical: an Ingress backend references its Service port **by name**, while `backendRefs` takes a `PortNumber` and cannot, so the number in the route and the port in the Service are kept in step by hand from now on.

The third check is a **pairing within one HTTPRoute object**, and the scope is deliberate: Gateway-API matching precedence is a property of a single route, so a redirect in a *different* route (a different hostname) says nothing about this one. Precedence also makes the pairing a question of existence rather than order — an `Exact` path match outranks every prefix match, so the redirect rule wins wherever it is emitted, and the rules' length-descending order stays a rendering convention. `freesailarr` is the first chart it applies to — **and, after group 3, still the only one.** The line that used to stand here said `bulwark` was the second `rewrite-target` chart and would meet the same check when group 3 landed. Group 3 landed and it does not: **`bulwark` has one Ingress object, no `rewrite-target`, and cannot have one** — `NEXT_PUBLIC_BASE_PATH` is compiled into its bundle, so it cannot be served under a sub-path at all and its route is one rule at `/` with no `URLRewrite` filter. Every occurrence of the string `rewrite-target` in that chart is an explanation of *why it does not help*, and #114's measured table arrived at "`bulwark`, `freesailarr`" by grepping for the string rather than for the annotation. The check still runs over all charts for the reason below — the mistake arrives by copy into a chart whose own diff nobody reads for it — not because a second chart is known to need it.

Three things to know before touching it:

- **Helm is the YAML parser**, through `ci/yaml2json`, for the same reason as the selector-match test above — the runners have neither `pyyaml` nor `yq`, and a parser written in `awk` fails by reading nothing and reporting nothing.
- **What is deliberately not checked**, named so the gaps are not mistaken for coverage. *That traffic flows* — no Gateway controller runs in CI and none will, so `Accepted: True` is not obtainable; that claim is simply not made anywhere in this pipeline. *That any redirect is ever served* — the pairing check proves the rendered **object** carries a `RequestRedirect` rule on the bare prefix and nothing more: whether an implementation honours the `statusCode` or declines it and fails the whole route is outside what a render can see. *Mutual exclusion with the Ingress* — each chart aborts its own render when `ingress.enabled` and `httpRoute.enabled` are both true, which is where it belongs: this test renders one configuration per chart, and a rule here would have to assume the switch is chart-global, which may not survive the multi-app charts of the sweep. *Cross-namespace and non-Service backends*, and a rule with no `backendRefs` at all (legal for a filter-only rule) — all three are **reported as "not checked", never counted as passing**. The redirect rule the pairing check demands is itself backend-less, so a chart that satisfies that check also earns one "not checked" line; both verdicts are correct at once.
- **Two anti-vacuum guards**, because the convention ships `httpRoute.enabled: false` and this test would otherwise be green over nothing. `ci/httproute/<chart>.yaml` is a render-only overlay applied **last** whose one job is to put a chart into Gateway mode — and a chart that *has* one and still renders no route **fails**. On top of that the run fails if **no assertion at all** held anywhere. `cyberchef` is exercised through its ordinary install-test fixture (which installs in Gateway mode, see below), every other converted chart through its overlay. For a multi-app chart the overlay also has to enter every shape the template has: `ci/httproute/freesailarr.yaml` moves one app onto a host of its own so that the shared-host route *and* the per-app route are both rendered.

Measured in CI (run 719, the pilot): **30 charts, 4 assertions held**, whole job 13 s including checkout and the Helm setup, no cluster. Group 1 took that to 20 assertions over 10 charts and group 2 to 36 over 15; the prefix/redirect pairing of Gitea #118 adds the first check since the pilot rather than a chart, which brought it to 37 over 15 — one assertion for `freesailarr`'s `/qbittorrent`, plus one more "not checked" line for the backend-less redirect rule it demands. **Group 3 takes it to `56` assertions over `21` charts**, which is the sweep finished except for the frozen `rss-bridge`. Group 3's 19 come from six charts and three of them contribute more than one route: `keycloak` 4 (public and admin routes), `matrix-synapse` 4 (homeserver and delegation routes), `homeassistant` 5 (two routes, three rules — its overlay is the only one in the repo that has to *supply* hostnames, since the chart has no `ingress.domain`), `bulwark` 2, `stalwart` 2, `ytdlp-web-player` 2. **The realm hostnames of Gitea #122 add `4`**: `keycloak`'s share goes from 4 to 8, because its two realm routes each earn one `parentRefs` assertion and one `backendRefs` assertion on their prefix rule, plus one "not checked" line each for their backend-less redirect rule (`keycloak: ok (8 assertion(s) held, 2 not checked)`). Measured as `60` over `21` charts on the branch that added them, which did **not** contain the `rss-bridge` conversion — the two landed in parallel, so the **delta** is the durable figure here and the running total is whatever the first CI run on `main` prints after both. No new *check*: the two realm routes are objects, and the prefix/redirect pairing of #118 deliberately does **not** fire on them — it is scoped to rules that strip a prefix with a `ReplacePrefixMatch` rewrite, and a realm route strips nothing, it redirects the bare root of a hostname that serves one realm. The fail path was exercised rather than assumed: a Service renamed in the route, a port the Service does not expose, a port left out entirely, an emptied `parentRefs` with the chart's own guard disabled, an overlay that renders no route, a run with zero assertions, and — for the pairing — both the redirect rule deleted outright and the redirect pointed at a path other than the one stripped; each produced a `FAIL` naming both sides and a non-zero exit.

### Install-test (PR pipeline stage 2)

For every chart a PR changes, `ci/run-install-test.sh <chart>` runs against a throwaway k3d cluster: it applies the stub `OnePasswordItem` CRD plus the chart's fixtures, `helm install`s the chart with its CI values, and waits for all Deployments/StatefulSets to roll out (per-chart timeout). Since probes and security hardening are active on all workloads, "Ready" means the app actually answers. On failure the job dumps pod descriptions, events and container logs.

The job itself is the reusable workflow **`install-test.yml`**; `validate-charts.yml` calls it with the changed charts, `install-test-all-charts.yml` with all of them. Everything about *how* a chart is installed is in that one file, deliberately — see "All-charts install-test" below.

Fixtures live at the **repo root** under `ci/` (deliberately *not* inside the chart directories, so they are neither packaged into the chart tgz nor trigger the publish workflows on merge):

- `ci/common/` — fixtures applied for every chart (schema-less stub CRD for `OnePasswordItem`)
- `ci/gateway-api-crds.sh` — the **real** Gateway-API CRDs (standard channel, pinned, Renovate-tracked), applied only for a chart whose render actually contains an `HTTPRoute`. **Deliberately not a stub and deliberately not in `ci/common/`.** A schema-less stub would make the exercise worthless here: the API server would accept a malformed `parentRefs`, a port written as a string, a path type that does not exist, and the install-test would prove only that *something was created* — for `OnePasswordItem` that is the right trade (nobody is testing the operator's API), for an `HTTPRoute` the schema **is** the point. Applied with `kubectl apply --server-side`, and the reason is measured: client-side apply stores the whole object in the `last-applied-configuration` annotation, annotations total at most 262144 bytes, and the HTTPRoute CRD of v1.6.3 is 244792 bytes as JSON — 93 % of the limit, with every release adding fields. Client-side happens to fit today and is one upstream release away from failing with `metadata.annotations: Too long` on a Renovate PR that has nothing to do with it. Downloaded rather than vendored, like Trivy's tarball and `kubectl`: 1.1 MB of generated YAML in the repo would go stale invisibly, and the price of the fetch is named — if the asset disappears, an unrelated PR goes red, loudly
- `ci/<chart>/test-values.yaml` — values for the CI install (required for every installable chart)
- `ci/<chart>/fixtures.yaml` — dummy secrets with in-cluster connection data, throwaway infrastructure (postgres/redis/dex), static hostPath PVs/StorageClasses where the chart pins a storage class or needs RWX, and seed jobs (e.g. `synapse generate`)
- `ci/<chart>/fixtures.env.yaml` — fixtures that need a value which must not be committed. The file holds `${VARIABLE}` placeholders and nothing else; `ci/run-install-test.sh` renders it with `envsubst` and pipes the result straight into `kubectl` (never printed). Substitution is restricted to the names from `REQUIRED_ENV`, so an undeclared variable is never filled in and no unrelated `$…` in the manifest is touched
- `ci/<chart>/settings.env` — optional overrides: `INSTALL_TIMEOUT` for slow starters, and `REQUIRED_ENV="VAR_A VAR_B"` to declare the variables `fixtures.env.yaml` needs. **A missing or empty declared variable fails the install-test loudly, naming the variable — it never skips the chart.** Skipping would restore exactly the green-but-meaningless signal this mechanism exists to remove; the one legitimate case of absent secrets, a PR from a fork, belongs in an `if` on the workflow job rather than in the script
- `ci/excluded-charts.txt` — charts that cannot run in CI, one per line as `<chart> <justification>`. Currently: `satisfactory` (multi-GB image + 10GB+ Steam download), `template-chart` (scaffold with placeholder image). The justification is mandatory: a line carrying only a chart name is deliberately **not** an exclusion, because a bare name would drop a chart out of CI with nothing to review. **An excluded chart gets no install-test job at all** — see below

`freesailarr` is the only chart using `fixtures.env.yaml` today: gluetun needs a real Wireguard key, or it never completes a handshake and no app container behind its kill switch becomes Ready. The install-test job loads the key from 1Password Connect for that chart only, using the same pattern as the shared publish block — `steps.<id>.outputs.*` into the step's own `env:`, quoted, with `:?` guards that name the missing `op://` field before the cluster is even created. The key in that item is a **CI-only Mullvad key**, deliberately not the production one: Mullvad ties an address to a key and routes the tunnel to the endpoint it saw last, so a shared key would make every PR run pull the tunnel out from under the running stack — with the symptom in the cluster and the cause in a pull request.

**An excluded chart is dropped from the matrix, not skipped inside its job (#231).** It used to get a job that ran the script, printed `SKIP` and exited 0 — and GitHub knows only the exit code, so the check list showed `Install-test satisfactory ✓` for a chart whose image was never pulled, indistinguishable from one that really came up in the cluster. `detect-changed-charts` now removes the excluded charts before the matrix is built, so no job exists for them and nothing claims they passed. Because an absence on its own says nothing, that job writes every excluded chart of the PR **with its documented justification** into the job summary — a `Skipped` would only say "not run", and why is the question a review actually asks. The list is parsed in exactly one place: `ci/run-install-test.sh --list-excluded` emits `<chart><TAB><reason>` and the workflow calls it instead of re-implementing the format in YAML, since a second parser would be a second chance to select different charts — silently, and in either direction. The script keeps its own `SKIP` branch for local runs. What cannot be done instead: deciding per chart in an `if` on the install-test job, to get a visible `Skipped` entry rather than an absent one. GitHub rejects it with `context "matrix" is not allowed here` — `jobs.<id>.if` is evaluated **before** the matrix is expanded, so the `matrix` context does not exist there, and the whole workflow ends in failure with zero jobs created (measured on the closed PR #233).

Fixture images avoid Docker Hub where possible (`public.ecr.aws`/`ghcr.io`) to dodge runner rate limits. The job **blocks the PR** on failure — the stabilization phase of #22 ended when Renovate started automerging patch/minor updates, since a chart that no longer starts must not reach `main` and publish unattended. When adding a new chart, add its `ci/<chart>/` fixtures (or an entry in `ci/excluded-charts.txt`) — the install-test fails if both are missing. The script also runs locally against any existing cluster: `k3d cluster create t && ci/run-install-test.sh <chart> && k3d cluster delete t`.

### All-charts install-test (on demand)

`install-test-all-charts.yml` runs **the same job** as the PR pipeline — the same `install-test.yml`, the same pinned `kubectl`, the same fixtures — with a matrix over **every installable chart** instead of the charts a PR changed. `workflow_dispatch` only, **no schedule**: a daily run would spend compute on a question that changes rarely, and a red run nobody asked for is a red run nobody reads.

It answers a different question from the PR pipeline. The PR install-test guards against what a change breaks; this one asks whether the environment under an **unchanged** chart still holds. The install-test matrix follows the charts a PR touches, so a chart nobody has edited since the forge move carries a green history from a *different runner* — and that green says nothing about today. **Run it after a change to the runner, to `K3S_IMAGE`, to the pinned `KUBECTL_VERSION`, to the fixtures under `ci/` — and after a forge move.** It is also the answer for a PR that only changes the harness and therefore tests nothing by itself: run it once the change is on `main`.

Why it exists at all: the answer arrived once by accident. Gitea PR #48 corrected a *comment* in 21 charts, which made all of them "changed" and turned the run into the first all-charts install-test after the move — **19 of 20 matrix charts came up in 12m41s** in one job, and exactly one did not (`rss-bridge`, whose last green run was from a GitHub runner; Gitea #51). Worth having, and nobody had planned it.

- **Input `charts`** — space- or comma-separated chart names, empty means all. That is what lets a single chart be tested against the runner, which is what Gitea #51 needs. A name that is not a chart directory, or one listed in `ci/excluded-charts.txt`, is a **loud error**: someone typed it, and a green run over zero charts would be the worst possible answer. An empty matrix fails the run for the same reason.
- **One throwaway cluster per chart, no reuse.** Reuse would save time and change what the test proves — leftovers from the previous install, a warm image cache, fixtures already applied. "Came up from nothing" is the statement.
- **Exclusions come from the one parser**, `ci/run-install-test.sh --list-excluded`, and the excluded charts appear in the job summary **with their justification**, exactly as in the PR pipeline.
- **The result is written down.** Every matrix leg records its own outcome next to the four facts that decide what the outcome means — date, runner label, `K3S_IMAGE`, `kubectl` version — and a final job records the verdict over the whole matrix. A green run nobody notes is unknown again next time. Per leg and not as one table, because a matrix leg only ever sees its own job summary.
- **Expected runtime: 20 charts in 12m41s** (Gitea PR #48, one job, all legs concurrently on one Docker daemon). The figure is there so a later deviation is visible as one; compare it against the chart count the run prints in its own summary — 24 at the time of writing, and one more with every new chart. The job timeout is 30 minutes per leg and lives in `install-test.yml`, i.e. it is the same for both callers on purpose.

### SonarQube analysis (after the merge, and not a gate)

`sonar.yml` sends the whole tree through a SonarQube analysis. The workflow itself is the shared block [`sonar-analysis.yml`](https://git.jk-dev.de/workflow-util/pipelines/wiki/sonar-analysis) in `workflow-util/pipelines`, bound to `@v4`; the only thing on this side is `sonar-project.properties` in the repo root, holding the project key `JLab-Helm-Charts`. **That key is read off the existing project in SonarQube, not derived** — it is the one the maintainer set up, and a guessed key (`jlab-cluster_helm-charts`, reasoned from the instance serving two owners) is what made the first run fail. The file carries **no `sonar.projectName`** on purpose: once the project exists, the display name belongs to the server, and a name sent with every analysis would overwrite a rename there silently.

**What is analysed — measured, not taken from a language list.** The first run (run 375) loaded exactly **two quality profiles, `json` and `yaml`**; the sensors that did work were **IaC Kubernetes** (3.4 s) and the **secrets detection** (`TextAndSecretsSensor`) over all 391 indexed files.

| What is in this repository | Analyser | Runs? |
|---|---|---|
| 238 YAML files and 26 Helm template directories | IaC Kubernetes, YAML | yes |
| 26 `values.schema.json` | JSON | yes |
| the whole tree (391 files) | Secrets | yes |
| the 5 shell scripts under `ci/` — the whole CI logic | Shell | **no** |
| the 30 workflow files | GitHub Actions | **no** |

**The two missing rows are not a configuration mistake, and nothing in this repository can fix them: analysers come from the *server*, not from the scanner.** The language list the earlier version of this table was built from describes the newest SonarQube release; this instance does not have the Shell and GitHub Actions plugins. The consequence is worth stating plainly rather than leaving it to be rediscovered — **the files carrying the actual logic, the five `ci/` scripts and the thirty workflows, are not analysed at all.** What runs is IaC over the charts plus secret detection over everything. Whether the instance gets upgraded or gains those plugins is a decision for whoever runs it, not something this repository settles; `shellcheck` and review remain the only check on the scripts until then.

The secrets row is the valuable one that *does* run. **The premise of this repository is that no credential ever lands in it** — every secret is a 1Password `OnePasswordItem` reference — and until now nothing checked that premise. What it actually reports is still unknown: run 375 built the report (603 kB) but never uploaded it, because the upload is where the wrong project key failed.

**It is not a gate, and that is a decision.** The SonarQube Community Build analyses the **main branch only**; there is no pull-request analysis to hook into, so the trigger is a push to `main` and **a finding appears after the merge**. At that point there is nothing left to block, which is why the shared block carries no quality-gate step: a red gate behind the merge would only colour `main` red without having prevented anything, and a permanent red is a thing people learn to ignore — taking the credibility of `validate-charts` with it. A gate becomes the right thing under one of two conditions: pull-request analysis is available, or a deployment hangs off the workflow. Until then, read the findings in SonarQube; nothing in this repository turns red because of them.

**No exclusions, for the first round.** Including the Helm templates, which are not valid YAML because of `{{ }}`. What the Helm and YAML analysers make of them is only learnable by letting them run over them once, and excluding them up front would discard that answer along with the bulk of the repository. If the first run produces noise genuinely not worth reading, the exclusions go into `sonar-project.properties` with the finding they answer named next to them — not before.

**Prerequisites**, both on the instance and neither in a file here: the repository **variable `SONAR_HOST_URL`** (a variable and not a secret on purpose — the server address is not a credential, and as a variable it stays readable in the log, which is what makes a wrong one diagnosable) and the **secret `SONAR_TOKEN`**, which reaches the block through `secrets: inherit`. The block guards both and names the setting and its place when one is empty. A *missing* `SONAR_TOKEN` never gets that far: the block declares it `required`, and a missing required secret fails job **creation** on this instance, so the run would show no steps at all — the same quirk as everywhere else here.

**Three things about this path were unmeasured, and run 375 answered all three — positively.** `node24` **is** in the runner image, so the JavaScript scan action runs. The scanner download **gets through**: ~59 MiB of archive plus its signature in roughly three seconds, so the MTU failure that once killed a 17 MB download on this runner (see "What a GitHub runner had been providing quietly") does not repeat at five times the size. And a repository **variable does reach a workflow called from another repository** — the block's guard passed and the scanner started, which it could not have done without `vars.SONAR_HOST_URL` arriving. That last one also settles the identical question the publish route carried for `OP_CONNECT_HOST`: same forge version, same mechanism, a repository variable read inside a workflow called from another repository. What the publish route still has unproven is the upload itself, which has never run. `workflow_dispatch` stays on the hull for the retries: without it every attempt costs a dummy commit on `main`.

## Dependency updates (Renovate)

Renovate runs as the shared `jlab-renovate-bot` instance on the Gitea side, which also serves the other repositories there — see "Renovate on Gitea" below. **There is no Renovate workflow in this repo, and none should be added.** (The `jlab-renovate` GitHub App that used to do this is switched off along with the rest of GitHub Actions.) What updates is unaffected either way: the config is **`renovate.json5`** in the repo root (JSON5 for the inline reasoning; requires Renovate >= 41 for `managerFilePatterns`/`customType`). There must be exactly one config file — do not add a second `renovate.json` or `.github/renovate.json`.

What Renovate keeps current:

| What | How |
| --- | --- |
| Chart app versions (`appVersion:` in `Chart.yaml`) | One `customManagers` entry per chart, hard-coding the image repository |
| Sidecar/init images in `values.yaml` (nginx, guacd, busybox, alpine) | Built-in `helm-values` manager |
| **All** images of `freesailarr`, including its main one | Built-in `helm-values` manager — the chart pins every tag in `values.yaml` and has no app-version block. One exception: `mediathekarr` is pinned by **digest** (its tags carry no version) and has its own `customManagers` block. All ten arrive **grouped into one PR**, see "The one grouped chart" below |
| GitHub Actions | Built-in `github-actions` manager |
| Tool versions pinned as workflow env vars (`TRIVY_VERSION`) | `# renovate: datasource=… depName=…` comment above the line |

Each chart renders its main image as `<repo>:{{ .Chart.AppVersion }}` — repository in the template, version in `Chart.yaml`. Renovate cannot join two files, so the mapping lives in `renovate.json5`, one block per chart. It is deliberately *not* a `# renovate:` comment inside each `Chart.yaml`: that would touch all chart directories and force a version bump plus a publish per chart for a pure comment change. Charts whose tags carry a flavour or date suffix (`apache-tika`, `minecraft-server`, `rss-bridge`, `searxng`, `teamspeak-server`) override `versioning` in their block; the reasoning sits next to each one.

One rule those overrides share, learned from `searxng` and worth knowing before writing the next one: **a capture group is a claim about how the value is compared.** Renovate's regex versioning parses `major`, `minor`, `patch`, `build` and `revision` with `Number.parseInt` (`modules/versioning/regex/index.js`, `_parse`), so a non-numeric commit hash captured as `build` becomes whatever digits lead it — `9fea41204` is `9`, `777ba8fa4` is `777`, and a hash starting with a letter is `NaN`. Measured on renovate 41.173.1 and 44.48.0, that makes Renovate offer an image pushed *earlier the same day* as an update to a newer one — a downgrade wearing the clothes of a routine date bump, which is exactly what a review waves through and automerge then carries to `main`. Parts of a tag that carry no ordering belong **outside** every group: the anchored pattern still requires them to be there, they simply take no part in the comparison. `prerelease` is not the way out either — it sorts correctly but marks every tag `isStable === false`, which hands the whole chart to `ignoreUnstable`. `freesailarr` has no **app-version** block on purpose — see the images exception above; a block would either duplicate the `helm-values` updates or rewrite an `appVersion` that selects no image. It does carry one `customManagers` block, for `mediathekarr`: that image is pinned by digest because its tags carry no version (`latest` is a 2024 build, `beta` moves), and the built-in `helm-values` manager reads only `repository`/`tag` and skips it (`skipReason: invalid-value`), so the two cannot collide. Digest rules need the same guard as versioning rules and for a sharper reason: while the pin is current, a matching rule and a broken one produce the identical empty result. Verified with two dry-run fixtures — the real digest yields no update, a deliberately stale one yields exactly one `digest` update back to the pinned digest.

A consequence of pinning every tag in `values.yaml`: `ci/renovate-bump-chart.sh` sees an unchanged `appVersion` and bumps the chart **patch**. That holds for all ten images alike, the digest-pinned one included — none of them is special — and it is the intended outcome, since `appVersion` here is the chart's own marker rather than a tracked upstream version.

**An open point this leaves, recorded rather than solved.** That automatic patch is also a classification, and the convention exempts a "plain app-version bump" from the wiki obligation — so an image update that quietly changes a *default* is the one case where nothing reviews it and nothing documents it. It has already happened once: the `sonarr` 4.0.19 bump carried upstream's removal of the hard-coded trusted proxy networks, and it reached `main` unremarked (see [Lessons learned](https://git.jk-dev.de/jlab-cluster/helm-charts/wiki/Lessons-Learned)). Whether the convention should change is a separate decision and deliberately not proposed here.

Two dependencies are constrained beyond their versioning. **nginx** (the Synapse delegation sidecar) is limited to **even minors** via `allowedVersions`. nginx numbers its release branches by minor — even is stable, odd is mainline — and the chart belongs on stable. Patch updates inside the stable branch still flow through normally. The constraint tolerates a variant suffix (`1.30.4-alpine`) on purpose: docker versioning only ever offers candidates carrying the current tag's suffix, so a suffix-blind constraint would silently freeze nginx the day the sidecar moves to a slimmer base image — a switch to `1.30.4-alpine` that #84 has commissioned a check on — with no error and no warning.

**linuxserver/qbittorrent** (freesailarr) excludes two tags by name, `14.3.9` and `20.04.1`. They are leftovers from earlier linuxserver tagging schemes, and docker versioning sorts both above every real release, so Renovate offers them as major updates — PRs #152 and #153 were exactly that. They are named rather than cut off with an upper bound like `<10`, because a bound would eventually reject a genuine major, while two finished artifacts cannot grow and therefore cannot hide a future release. The same guard as for nginx applies and was run: a lookup from `currentValue 4.6.4` still offers `5.2.3` as a **major**, so the filter does not silently empty the update list. That fixture also shows the cost of leaving it alone — without the rule the very same lookup offers `20.04.1` and never mentions the real `5.2.3`, since Renovate fills the major bucket only once.

### The one grouped chart

`freesailarr` is the only chart whose updates are **grouped**: all ten of its images arrive as one PR per cycle (`groupName: "freesailarr stack"`), where every other chart in this repo gets one PR per image. For the others that is already the same thing — one chart, one image — so this is a rule about a ten-image chart rather than a change of policy. It is also the only `groupName` in `renovate.json5`, and it reverses what the comment above the app-version managers describes: the `helm-values` manager finding those ten images and proposing "one update per image". That sentence described what Renovate does, not what the chart wants.

**What it fixes, measured on 2026-10-12.** Eight of the ten images had an open Renovate PR simultaneously (#102 bazarr, #103 qbittorrent, #104 sonarr, #105 flaresolverr, #106 recyclarr, #107 prowlarr, #108 radarr, #111 seerr). Each edits `freesailarr/values.yaml` and — through the `postUpgradeTasks` hook — `freesailarr/Chart.yaml`, so the API reported `mergeable: false` for all eight: they rewrite the same `version:` line. One could merge; the other seven then needed a rebase each, and every rebase restarts the most expensive install-test here (a real Wireguard tunnel, ten containers). A sharper second symptom came with it: all eight carried `version: 3.1.1+up1.0.0 → 3.1.2+up1.0.0` while `main` already stood at `3.3.0+up1.0.0`, because the hook derives everything from the **base commit** and those branches predated the Gateway-API rounds. A rebase repairs that, which is precisely the problem — nothing *fails* if the rebase does not happen first, so the repair depends on the bot reaching the branch before a human reaches the merge button. Grouped, there is one branch, one `version:` computed from one base, one publish and one install-test run.

**Four decisions worth keeping** (the reasoning sits next to the rules in `renovate.json5`):

- **Matched by file name, not by a list of images.** What causes the collision is "touches `freesailarr/Chart.yaml`", true of every dependency Renovate finds in that `values.yaml` regardless of manager or datasource. A `matchPackageNames` list would be a second place to keep the chart's image set current, and its failure mode is silence — an image added later never joins the group, with no error (#158: a valid rule is not a matching rule).
- **`mediathekarr` is in the group** although it comes from its own `customManagers` block: its digest PR edits the same two files and blocks the others identically. `digest` is in `matchUpdateTypes` for exactly that, and its own block is untouched — a group decides which *branch* an update lands on, not how it is found.
- **`gluetun` and the `busybox` init image are in too.** Neither "gluetun is the tunnel" nor "busybox is only an init container" has anything to do with the blocking, and a grouped PR hides nothing: the review still sees every changed line, just not eight times over.
- **Majors stay out**, via `matchUpdateTypes: ["patch", "minor", "digest"]`. An app major must be taken stepwise in its own PR, and `separateMultipleMajor` splits a multi-major jump to produce that sequence; a group throwing a major in with seven patches would undo both.

**It takes two `packageRules`, and that is not cosmetic.** Renovate refuses `matchUpdateTypes` and `separateMinorPatch` in one rule — `packageRules cannot combine both matchUpdateTypes and separateMinorPatch`, a config-validation error — which is consistent, since `separateMinorPatch` is what *decides* the split a rule would then be selecting on. So one rule groups the non-major update types and a second sets `separateMinorPatch: false` for the same file. Without that second rule the group still arrives as **two** PRs: verified by dry-run (`--platform=local --dry-run=full`, real registries) against a checkout holding those eight updates, they landed on `renovate/freesailarr-stack` (the three minors) and `renovate/patch-freesailarr-stack` (the five patches) — note that `separateMinorPatch` **prefixes** the branch rather than suffixing it, which makes the second name easy to miss in a branch list. With both rules, one branch carries all eight. The global `separateMinorPatch: true` keeps its reason for every other chart; "one red install-test must not block unrelated updates" simply does not apply within a single chart, where the updates share a pod, a chart version and a publish.

What this cannot show from inside the repo: Renovate runs as the shared instance outside it, so a valid config and a correct local lookup are the evidence available — not the behaviour of the next real run.

Not managed: `ci/**` (install-test fixtures pin throwaway infrastructure to major lines on purpose) and `template-chart/**` (placeholder image). A consequence worth knowing: `ci/matrix-synapse/fixtures.yaml` pins a Synapse image that Renovate will not follow — bump it by hand when the chart moves a major.

### The version-bump hook

Renovate only rewrites `appVersion:` or an image tag; it knows nothing about `version: <chart-semver>+up<app-version>`. Left alone, every chart PR would fail the `build-chart` version check. `ci/renovate-bump-chart.sh <chart-dir>` closes that gap and is wired up as a `postUpgradeTasks` command:

- **app major changed** → chart major (`6.0.0` → `7.0.0`), matching "runtime behavior changes" in the semver conventions
- **anything else** (plain app update, sidecar image) → chart patch
- the `+up` suffix is recomputed from the new `appVersion`, dashes normalized to dots

Everything is derived from the base commit, so the script is idempotent across the multiple updates a single Renovate branch can carry.

> **One-time setup outside this repo:** `postUpgradeTasks` only runs on self-hosted Renovate **and** only when the command is allow-listed in the *instance* config:
>
> ```yaml
> allowedCommands: ["^ci/renovate-bump-chart\\.sh"]
> ```
>
> Without that entry Renovate skips the task without an error and every chart PR stays red on the version check.

If the dependency dashboard reports `Failed to look up github-releases package …: no-result` for third-party repos, that is the instance's github.com credentials, not this config. The instance needs a read-only PAT as `GITHUB_COM_TOKEN` for github.com datasource lookups; its own platform token is a Gitea token and is worth nothing there.

### Renovate on Gitea

**Do not add a Renovate workflow to this repository.** It is the obvious thing to reach for — Gitea has no app model, so a scheduled job in the repo looks like the only way to run the bot — and it is wrong here: a **shared `jlab-renovate-bot` instance** runs outside this repository and has it enrolled. A second runner against the same repo produces duplicate PRs and two schedulers rewriting each other's branches, which is not a safety net but a fault. This paragraph exists so the next reader who finds no workflow knows why instead of building one.

The instance is outside this repository, which has two consequences worth writing down:

- **It reads our `renovate.json5` and the per-chart rules do fire.** Measured, not assumed: its first run here opened PRs for `searxng` and `gotenberg` plus the dependency dashboard. Those two are `customManagers` targets of ours — no built-in manager finds an `appVersion:` in a `Chart.yaml` — so the config is demonstrably in effect.
- **`allowedCommands` is already granted there, and the version-bump hook runs.** This was the expensive unknown: `postUpgradeTasks` only executes when the command is allow-listed in the *instance* config, and a missing entry is skipped **silently**, leaving every chart PR red on the version check. The proof is in that searxng PR's diff, which carries `version: 1.0.4+up2026.9.11.ffe96f8a6` → `1.0.5+up2026.9.13.96479fff6` alongside the `appVersion` change. Nothing but `ci/renovate-bump-chart.sh` writes `version:`. The allow-list is not visible from this repo, though, so if it is ever dropped the symptom here will be red chart PRs with no explanation.

One item is easy to miss and is named in "The version-bump hook" above: the instance needs a `GITHUB_COM_TOKEN` for github.com datasource lookups. Every `github-releases`/`github-tags` lookup and the whole built-in `github-actions` manager resolve against github.com, the platform token is a Gitea one and worth nothing there, and unauthenticated lookups hit the rate limit inside a single pass. The symptom is a `no-result` line on the dependency dashboard, not an error.

### PR policy

The config is aligned with the one in `cluster-config`, because the same `jlab-renovate-bot` instance serves both and two repos that behave differently under one bot is a source of surprises rather than of safety.

- **Automerge is global** (`automerge: true`, `automergeType: "pr"`, `platformAutomerge: true`) — every update type, **majors included**. The safeguard is not the absence of automerge, it is the branch protection: `main` requires a **review** and requires `validate-charts` to pass. `platformAutomerge` arms the PR on the forge, and Gitea merges it when *both* conditions are met, so an armed PR without a review simply sits there. This is the setting the older `platformAutomerge: false` comment in `renovate.json5` named as the better one "once required checks exist on `main`" — they now do. Same reasoning as `cluster-config`, which retired its dashboard-approval step for it.
- **`minimumReleaseAge: "3 days"`** for `patch`, `minor` and `digest` updates. Renovate does not even open the PR before then. A merge here is never just a merge — a chart publish follows it straight away — so a release that gets yanked hours after it appeared must not reach the registry. Majors carry no age gate: they are held at the review, not by a delay.
- **Majors** arrive one line at a time (`separateMultipleMajor`): a v1 → v3 jump produces a PR for the latest v2 first and then one for v3 — the stepwise procedure the versioning section requires. `separateMinorPatch: true` likewise splits minor and patch into separate PRs, so one red install-test does not block unrelated updates. **It has one exception**, `freesailarr`, where it is overridden to `false` — see "The one grouped chart" below for why that separation is wrong inside a single chart.
- **No PR caps** (`prConcurrentLimit: 100`, `prHourlyLimit: 0`), as in `cluster-config`. The known cost, written down rather than discovered: the first run after a quiet week can open many PRs at once, and every chart PR starts a k3d install-test — which on a Gitea runner queues inside one job rather than fanning out (see "Where this repo lives"). That queue is the accepted trade for not having updates trickle in four per hour.
- **`config:best-practices`** is in `extends`. Two of its members matter here: `helpers:pinGitHubActionDigests` moves the workflow files from `actions/checkout@v7` to SHA pins (wanted), and `docker:pinDigests` is switched off again for `custom.regex` and `helm-values` — an `appVersion:` is a tag by construction (`<repo>:{{ .Chart.AppVersion }}`, and the `+up` suffix is built from the same string), so a digest has nowhere to go. Sidecar digests in `values.yaml` *could* be pinned; that is a separate decision, since each digest change would be a chart patch plus a publish.
- The manually dispatched `change-app-version.yml` still exists for one-off bumps, but the normal path for app updates is now a Renovate PR.

## Creating a new chart

The wiki carries the same procedure as a worked checklist: [Creating a new chart](https://git.jk-dev.de/jlab-cluster/helm-charts/wiki/New-Chart). A new chart also needs a wiki page of its own before its PR is merged.

1. Copy `template-chart/` to `<chart-name>/` and replace every `xxx` placeholder: template file names (`xxx.deployment.yaml` → `<app>.deployment.yaml`), `Chart.yaml`, the `values.yaml` keys, the `.Values.xxx.*` references inside the templates, **and both the `xxx.` helper prefixes and the values paths in `values.schema.json`**. Then walk the six points in "Values that must not lie" below — the schema and the null-safe fallbacks are copied with the template, but their content is chart-specific and wrong until you adapt it. The same goes for `templates/NOTES.txt`: it comes with the template as a worked example and is placeholder text until you replace it with what an operator of *your* app needs after the install (see "Everything else" below); its leading `{{/* … */}}` block explains the bar and is meant to be deleted.
2. Add `.github/workflows/publish-<chart-name>.yml` (copy an existing one, e.g. `publish-outline.yml`).
3. Add the chart name to the choice list in `.github/workflows/change-app-version.yml`.
4. Add a `customManagers` block for the chart's main image in `renovate.json5` (copy a neighbouring one, swap `managerFilePatterns` and `depNameTemplate`) — without it the chart's app version is never updated. Check the image's tag list first: if it carries a flavour, date or prerelease suffix, add a `versioningTemplate` with a comment saying why. **Floating tags belong outside the pattern**: `ytdlp-web-player` anchors its versioning regex on the `v` prefix precisely so `latest` and `stable` — both of which move — can never be selected.
5. Only if the chart needs a **dependency**: read "Chart dependencies" in the style guidelines first, it is a bar rather than a formality. Nothing has to be added to the pipelines — `ci/helm-dependencies.sh` already runs everywhere and keys off the `dependencies:` block — but the subchart's own updates then depend entirely on Renovate's built-in `helmv3` manager plus a chart-version bump, which is the part to write down at the pinned version.

Validate locally with:

```bash
helm lint --strict <chart-name>
helm template <chart-name> --set ingress.domain=example.com --set ingress.clusterIssuer=letsencrypt
```

(required values vary per chart)

For runtime verification (do the workloads actually start, do probes/security settings hold?), use a throwaway local [k3d](https://k3d.io) cluster: `k3d cluster create <name>`, apply a stub CRD for `OnePasswordItem` plus dummy secrets (and throwaway postgres/redis pods where the app needs them), `helm install` the chart, watch pod readiness/events, then `k3d cluster delete <name>`.

## Chart style guidelines

These guidelines are **binding for every current and future chart** in this repo. They are the consolidated end state of the conventions overhaul (#32, batches 0–4); `template-chart/` is the reference implementation. The wiki mirrors them as the [chart style guide](https://git.jk-dev.de/jlab-cluster/helm-charts/wiki/Chart-Style-Guide), with the incident behind each rule collected in [Lessons learned](https://git.jk-dev.de/jlab-cluster/helm-charts/wiki/Lessons-Learned) — change a rule here and update that page in the same PR.

### 1. Naming

- Resources are named `{{ .Release.Name }}-{{ .Chart.Name }}-<kind>` (e.g. `...-deployment`, `...-sfs`, `...-service`, `...-secret`, `...-configmap`, `...-ingress`, `...-pvc`).
- Template files are named `<app>.<kind>.yaml`; stateful apps use a StatefulSet in a `<app>.sfs.yaml` template instead of a deployment.
- **Documented exception — existing PVC names**: PVCs of already-published charts keep their historical names even where they deviate from the scheme. Renaming a PVC makes Kubernetes provision a new, empty volume, so a rename would orphan the existing data. New charts name PVCs `{{ .Release.Name }}-{{ .Chart.Name }}-pvc` per the scheme.
- StatefulSet renames are possible (data survives because the charts use standalone PVC templates, not `volumeClaimTemplates`) but only ship in a major-bump PR with a one-time migration note: `kubectl delete statefulset <old> --cascade=orphan` (and delete the leftover old pod) before upgrading.

### 2. Versioning

- `Chart.yaml` uses `version: <chart-semver>+up<appVersion>`; the `+up` suffix matches `appVersion`, with dashes normalized to dots (e.g. `0.0.36-beta` → `+up0.0.36.beta`). `appVersion` is the **exact image tag** of the main app image.
- Semver rules for the chart part:
  - **Patch**: bug fixes without behavior or values-schema change; plain app-version updates.
  - **Minor**: additive, backward-compatible changes (new optional values with unchanged defaults).
  - **Major**: runtime behavior changes or breaking values changes (e.g. resource renames, probes, security hardening, values that become required).
- Any push to `main` touching a chart directory publishes that chart, so the version must be bumped in the same change.

### 3. Images

- The main app image tag comes from `{{ .Chart.AppVersion }}` — never hardcoded.
- **Documented exception — freesailarr**: the chart bundles ten independently versioned upstreams (nine of them enabled by default), so none of them is "the" app version. **No** tag comes from `appVersion`; every image is pinned as `<app>.image.repository` / `<app>.image.tag` in `values.yaml`, which is also what puts them under the built-in `helm-values` manager. Its `appVersion` is the chart's own version marker, freely chosen and tied to no upstream — raised by hand when a new edition of the chart is marked as a whole. Consequently the chart has **no app-version** block in `renovate.json5` and is **not** in the `change-app-version.yml` choice list. `mediathekarr` adds a second deviation on top: its tags carry no version at all, so it is pinned by `image.digest` as well and is the one image in this chart with a `customManagers` block — a `repository`/`tag` pair alone would leave it without any update path.
- **Documented exception — minecraft-server**: the only chart whose main image tag can be overridden, through `image.tagOverride` (#158, empty by default — the tag still comes from `appVersion`). `appVersion` is the tag and therefore fixes the JVM, and since chart 5.0.0 only the **newest** Java line is maintained out of this one directory; the older lines stay published as frozen chart versions (java8 and java17 at 4.0.0, java21 at 4.1.0) and never receive another change. Without the override, running an older JVM would mean installing a frozen chart and giving up every later fix — so the line is a decision at rollout time rather than a property of the chart. The override is named `tagOverride`, not `tag`, because it decouples the running image from the chart version, and it is deliberately outside Renovate's reach (`renovate.json5` tracks the `appVersion` in `Chart.yaml` and nothing else). The chart checks the pairing itself: the Java line of the **effective** tag against `minecraft.version`, failing the render with both values named — a mismatch used to install silently and only break the container afterwards.
- Auxiliary images (sidecars, init containers, secondary workloads — e.g. the delegation nginx in matrix-synapse, guacd in patchmon, busybox init/consume images in paperless-ngx) are pinned via values instead of being hardcoded in templates, and their pinned versions are reviewed alongside every app-version update round.

### 4. Ingress and Gateway API

- `ingress.domain` and `ingress.clusterIssuer` (cert-manager) are `required` values **without defaults** — every installation sets both explicitly; they are the only values needed for a default install. `clusterIssuer` is required **only while `ingress.enabled`**: in Gateway mode the chart renders no certificate, and a value that is demanded and does nothing is precisely the defect guideline 12 exists to remove.
- `ingress.className` selects the ingress class and defaults to `nginx`.
- `ingress.annotations` takes optional extra annotations that are merged with the cert-manager `cluster-issuer` annotation the chart always sets.
- The TLS secret is named `tls-<release>-<chart>-ingress`.
- **A set of objects publishing ONE backend under hostnames the chart does not own is the third admissible shape, and it buys its secret name and its issuer per entry (Gitea #122).** Where a chart's app serves several tenants from one deployment and each wants a hostname of its own — `keycloak`'s realms are the case this was written for — the hostnames come as a values list and the chart renders one object per entry. Two convention rules do not survive that, and both deviations are **required**, not merely tolerated: the TLS secret name is a per-entry value with **no** default, because these hostnames already have certificates and several entries typically share one (a wildcard for a zone), so a derived `tls-<release>-<chart>-...` name per entry would ask cert-manager for a second certificate for a name that is already covered — one ACME order per hostname, for nothing; and the cert-manager annotation becomes **optional per entry**, because cert-manager keys a Certificate by the secret it writes, so of several objects naming one secret exactly one may carry it. That is the same deliberate omission the qBittorrent object in `freesailarr` makes, for the same reason, and it is why the secret name is required rather than defaulted: a chart that invents the name cannot also leave the issuing to somebody else. Annotations are per entry too, since the set is not uniform across the hostnames. Object names carry the **list index**, for the reason spelled out in the homeassistant bullet below — a hostname is not a dependable object-name fragment, so reordering the list renames and recreates the objects — and the entries render in **both** routing modes — a mode-asymmetric feature is not admissible, which is also why the per-entry `parentRefs` exists as the counterpart of the per-entry class.
- **Documented exception — homeassistant, and it extends to the Gateway mode**: keeps its upstream-style flexible ingress values (host/tls lists, no required values); only its `ingress.className` default follows the standard (`nginx`, clearable to omit the field). Because it has **no `ingress.domain`**, it has **no `httpRoute.hostnames`** either — the second chart without that key after `freesailarr`, and the schema rejects it outright so the absence is enforced rather than merely intended. The hostnames and the paths come from `ingress.hosts`, which is therefore read in **both** modes and is not an Ingress-only value, and the chart renders **one `HTTPRoute` per entry** of that list with one rule per path of the entry. One object carrying several hostnames would be wrong rather than merely different: an `HTTPRoute`'s rules apply to *every* hostname of the object, and per-host paths are this chart's whole point. `ingress.className`, `ingress.tls` and `ingress.annotations` are not read in Gateway mode. Route names carry the entry's **index**, not its hostname (a hostname is not a dependable object-name fragment), so reordering `ingress.hosts` renames routes and therefore recreates rather than edits them — harmless, since a route holds no state and no certificate, but worth knowing before it looks like a bug. Its `pathType` default is the awkward one; see the `ImplementationSpecific` rule in the Gateway-API bullets below.
- **A shared domain with sub-paths is the second admissible ingress shape.** Where a chart exposes several apps, they may either each take a host of their own or share one host and be told apart by path (`domain.tld/sonarr`, `domain.tld/radarr`, …) — one DNS name and one certificate for the whole bundle. The rules for the shared shape: the domain is a single value (`ingress.host`), each app's path is its own value defaulting to `/<app>`, the paths render in one Ingress object `<release>-<chart>-ingress` with the standard TLS secret `tls-<release>-<chart>-ingress`, and an app that carries a host of its own always wins and keeps its own object — so both shapes mix without extra values. An app that needs a per-path proxy annotation (`rewrite-target` is per **object**, not per path) gets its own object on the same host, repeating the same TLS secret name and **omitting** the cert-manager annotation, so that exactly one certificate is requested. Apps that cannot serve a base path must be rendered at `/` or refused with a `fail` — never rendered somewhere they provably cannot work.
- **The Gateway API is the second routing mode, and it is implementation-neutral by construction (Gitea #114).** A chart offers it through one values block, off by default:

  ```yaml
  httpRoute:
    enabled: false     # counterpart to ingress.enabled
    parentRefs: []     # the Gateway (name, namespace, sectionName) — a platform object
    hostnames: []      # empty = [ingress.domain]
  ```

  The rules, each of which is load-bearing:

  - **Only `HTTPRoute` with `parentRefs`.** No `Gateway`, no `gatewayClassName`, no certificate, and **no implementation-specific policy object** (`ClientSettingsPolicy`, `ClientTrafficPolicy` and their equivalents). That subset looks identical on nginx-gateway-fabric, Cilium, Envoy Gateway and Istio, and keeping to it is the whole reason the cluster's implementation can be chosen — or replaced — without touching the charts again. A policy object binds a chart to one implementation *and* to a CRD whose presence it cannot check. Class and certificate therefore move to the **platform**: they sit on the Gateway and its listeners, in `cluster-config`.
  - **A value, not auto-detection.** `.Capabilities.APIVersions` is unreliable at template time, and an `HTTPRoute` without the Gateway-API CRDs fails at install time — so a chart that guessed would sometimes render an object the cluster rejects and sometimes silently render nothing.
  - **Mutually exclusive with `ingress.enabled`**, enforced by a `fail` that names **both** value paths. Two paths onto the same Service under one hostname is a conflict at DNS level already; a migration runs over a **second hostname** (`httpRoute.hostnames` set explicitly while the Ingress keeps `ingress.domain`), never over parallel operation.
  - **`parentRefs` is mandatory in Gateway mode**, enforced by a `fail` on the empty list. A route without a parent attaches to nothing and is simply inert — created, schema-valid, rejected by nobody, reported by nothing.
  - **One source of truth for the hostname.** Empty `hostnames` means `[ingress.domain]`; two values that have to agree drift apart. `ingress.domain` is therefore read in Gateway mode too — it names the release's public hostname and is not an Ingress switch.
  - **`NOTES.txt` says which mode is active**, as `keycloak` already does for its cases, and names what the mode costs.
  - **What has no counterpart, and must be said rather than rebuilt.** The Gateway API has no annotation channel for controller behaviour. `rewrite-target` has a real equivalent (the `URLRewrite` filter in a rule); **`proxy-body-size` has none** — an upload limit is implementation-specific policy, so a chart that depends on one must print in its Gateway-mode `NOTES.txt` that the limit is **not enforced** and has to be set at the Gateway. Charts that *set or recommend* one carry that disclosure: `calibre-web-automated`, `outline`, `paperless-ngx`, `zipline` since group 2, `bulwark` and `matrix-synapse` since group 3. `garage` *depends* on one without setting it: its `ingress.annotations` comment tells the operator to set `proxy-body-size: "0"` because ingress-nginx's 1 MB default truncates S3 uploads. Measured while sweeping group 1 — "sets no behaviour-bearing annotation" is not the same question as "needs none", and the sweep screens on the second one. `freesailarr` is the mirror image of `garage` and was measured in group 2: it sets no body-size annotation and needs none itself, but `ingress.annotations` is the only lever an operator has for one (qBittorrent takes torrent-file uploads), so its Gateway-mode `NOTES.txt` says that an annotation set there is gone. The same applies to auth annotations in front of an app with no login of its own: in Gateway mode that escape hatch does not exist in the chart at all (`cyberchef`, `searxng`), and so does any annotation a chart documents as its only lever for a response header (`bentopdf`: `X-Robots-Tag`).
  - **Screen the LIVE object, not only the chart — group 3's lesson, and the generalisation of Gitea #118.** A chart can read as correct while the running system fills a gap of the chart with an annotation the **operator** set, and the Gateway mode drops `ingress.annotations`, so it drops the compensation with it. At #118 that was found *after* shipping, on `freesailarr`, where the annotation was a redirect nobody would have looked for in the chart. Group 3 measured it beforehand with read-only `kubectl` against the live cluster, and **four of its six charts carried such an annotation that the chart never sets**: `keycloak` (`proxy-buffer-size: 16k` and `proxy-buffers-number: 4`, on *both* of its Ingress objects), `matrix-synapse` (`proxy-body-size: 50m`, plus the `nginx.org/client-max-body-size` twin for the other nginx controller, plus a `configuration-snippet`), `stalwart` (the whole `enable-cors` block), `bulwark` (`proxy-body-size: 25m`). None of them appears in a `git grep` of this repository, because no template sets any of them — the only hits are the *documentation* that recommends two of them. **The measurement at the live object is part of the procedure, not a flourish**, and reading the chart alone is not a substitute for it.
  - **And read the annotation's directive before calling it load-bearing.** `matrix-synapse`'s live `configuration-snippet` looks like exactly what Synapse's `x_forwarded` setting depends on — client IP for rate limiting, and federation. It is not: `more_set_headers` is the headers-more module's **output**-header directive ("Replaces (if any) or adds (if not any) the specified output headers"), the request-side one being `more_set_input_headers`. Those headers therefore go on the **response to the client**; Synapse never sees them, and what it reads with `x_forwarded: true` is the *request's* `X-Forwarded-For`, which ingress-nginx supplies itself. So the scariest-looking loss in group 3 is not a loss at all, and its `NOTES.txt` says in as many words **not** to re-create it at the Gateway and not to read its absence as the cause of a rate-limiting problem. The real loss on that chart is the body size. A reading of documented semantics, stated as such — nothing A/B-tested a running Synapse.
  - **"No counterpart" is a claim to check against the pinned CRDs, and for CORS it is false.** `stalwart`'s live Ingress carries `enable-cors` with an allow-origin pointing at `bulwark`, the webmail chart in the *same group* — so losing it breaks a second chart, with a browser console error and nothing at all in the server's log. Measured by grepping the CRD set this repo pins (`ci/gateway-api-crds.sh`, gateway-api v1.6.3 standard channel): HTTPRoute's filters **do** carry a full `cors` field (GEP-1767) with `allowOrigins`/`allowMethods`/`allowHeaders`/`allowCredentials`/`maxAge`, and its documented level, verbatim from that CRD, is `Support: Extended`. `responseHeaderModifier` is in the same CRD and is **also Extended, not Core**. The chart still renders neither, for three reasons in order: it has **no CORS values** to render one from (the origin and the method list are the operator's, and the chart sets no CORS annotation in Ingress mode either, so a filter would exist in Gateway mode only — an asymmetry that is its own defect and its own piece of work); **Extended support is not support**, since a rejected route is rejected *whole* (the reasoning #119 reached for `statusCode`) and the observed failure is quieter still — Cilium accepts `filters.type: CORS` and forwards the preflight upstream anyway, i.e. `Accepted: True` with the behaviour absent; and **half a CORS is worse than none, with the translatable half being the useless one here** — the browser's preflight must be answered *by the gateway* (the CORS filter's own contract forbids passing it upstream) and a bare header filter cannot do that, while this installation's allowed headers include `Authorization`, which makes **every** request preflighted. Hence: configure CORS at the Gateway with that implementation's own support *before* moving the host, and test from the browser — `curl` sends no preflight, so it passes while the webmail fails.
  - **`rewrite-target` is the one place where the Gateway API is less machinery than what it replaces, not a translation of it.** `rewrite-target` applies per **Ingress object**; a `URLRewrite` filter applies per **rule**. A chart serving several apps under one host, one of which needs its prefix stripped, therefore needs a *second* Ingress object for that app — which then has to repeat the other object's TLS secret name, omit the cert-manager annotation so only one certificate is requested (conditionally, for the case where no other object renders), and match on an `ImplementationSpecific` regex path whose capture group the rewrite refers to. In Gateway mode all four disappear: one route, one rule per app, the filter on one rule, a plain `PathPrefix`. Measured on `freesailarr` in group 2, and after group 3 it is still the **only** chart here with a `rewrite-target`: `bulwark` was listed as the second one by #114 and by this file, and it has none — see the HTTPRoute-test section for why the string occurs in that chart four times without the annotation ever being set.
  - **`pathType: ImplementationSpecific` has no counterpart either, and it is a chart *default* in one place.** Ingress offers `Exact`, `Prefix` and `ImplementationSpecific`; the Gateway API offers `Exact`, `PathPrefix` and `RegularExpression`. The first two map cleanly. `ImplementationSpecific` means "ask the controller", so nothing preserves it — and it is the default in `homeassistant`'s `ingress.hosts`, the one chart keeping upstream's flexible host list. The rule group 3 settled on, and the split is not tidiness: on the path `/` it is rendered as `PathPrefix`, because every controller treats a root `ImplementationSpecific` path as a prefix match and the chart's own default therefore works in Gateway mode unchanged; on **any other path the render aborts**, because in ingress-nginx an `ImplementationSpecific` sub-path is a **regex** and `PathPrefix` is not one, so mapping it silently would change how the path matches while the render stayed green — the #118 class of defect exactly. `RegularExpression` is deliberately not offered as an escape hatch: its support level is Implementation-specific, which is what the first rule above forbids.
  - **A value derived from `ingress.enabled` is a second thing the sweep has to look for, and it is not visible in the Ingress template.** Gateway mode sets `ingress.enabled: false` *by construction*, so any value computed from it flips — and where it stood for "is there a TLS-terminating proxy in front" or "is this app published at all", the flipped value is wrong while nothing fails. Two finds so far, both fixed by one helper reading `ingress.enabled OR httpRoute.enabled` rather than by a new value:
    - `sure` (group 1), whose `RAILS_ASSUME_SSL`/`RAILS_FORCE_SSL` would have gone `false` with TLS terminated at the Gateway: insecure cookies, no HSTS, `http://` links, green pod. Helper `sure.publishedOverTls`.
    - `freesailarr` (group 2), whose `freesailarr.sharedIngressApps` was gated on `ingress.enabled` and is read by the `<APP>__SERVER__URLBASE` environment variables as well as by the routing objects. In Gateway mode that list would have come out **empty**, so Radarr, Sonarr and Prowlarr would have been told no base path while their routes served them under `/radarr`, `/sonarr`, `/prowlarr` — a UI whose assets and API calls are linked at the host root, behind a Ready pod. The two guards living in that helper (seerr/mediathekarr only at `/`, no duplicate paths on the shared host) would have gone unenforced in that mode for the same reason. Helper `freesailarr.published`.
    - `ytdlp-web-player` (group 3), the one this file had listed as still open, and the find with the sharpest consequence. Its `ytdlp-web-player.validate` guard — which refuses to publish the application and the bundled oauth2-proxy on the **same host** — was gated on `ingress.enabled`, so in Gateway mode it went silent. **And that guard is needed more in Gateway mode, not less.** In Ingress mode ignoring it fails loudly: ingress-nginx's admission webhook rejects the second Ingress and the Fleet bundle lands in `ErrApplied`. In Gateway mode **the webhook never fires** — it compares Ingress objects with Ingress objects, and an `HTTPRoute` is a different kind — so nothing rejects anything, both objects stay valid, and one hostname is served over two data paths that are *not* equivalent: the proxy's Ingress goes through **authentication**, the chart's `HTTPRoute` goes straight to the application. Which one a client reaches depends on what DNS resolves to, so it is an authentication bypass that comes and goes with a record, reported by nothing. Helper `ytdlp-web-player.published`, and the Gateway-mode abort carries its own message saying exactly that.

    All three are now closed, and the screen was run over all six charts of group 3 rather than only the suspected one.
  - **A subchart does not come along, and that has to be said where somebody will look for it.** `ytdlp-web-player`'s oauth2-proxy dependency has **no Gateway-API support** and this repo does not patch upstream charts (see "Chart dependencies"), so it renders an Ingress or nothing. In the *recommended* configuration — the proxy in the data path, owning the host, the chart's own routing object off — the object that publishes the release is therefore **the subchart's Ingress in both of the chart's modes**, and setting `httpRoute.enabled` there changes nothing about how the host is served. The chart's own route is for the proxy-disabled case or for the application on a host of its own, which is the documented unauthenticated bypass. Stated in the template header, in `values.yaml`, in the schema and in the proxy branch of `NOTES.txt`, because an operator migrating hosts would otherwise set the value and wonder why nothing moved.
  - **CI**: either the chart's install-test fixture installs in Gateway mode, or it gets a render-only overlay at `ci/httproute/<chart>.yaml` — otherwise `ci/run-httproute-match-test.sh` is green over nothing for it, which that test fails on. An overlay for a multi-app chart has a second job: it must enter **every** shape the template has, or the branch nobody exercised is the one that breaks. `ci/httproute/freesailarr.yaml` gives one app a host of its own so both the shared-host route and the per-app route render.

  Reference implementation: `template-chart/templates/xxx.httproute.yaml`. Pilot: `cyberchef`. Group 1 — the charts rendering one Ingress with no behaviour-bearing annotation — followed: `bentopdf`, `garage`, `patchmon`, `rallly`, `searxng`, `sure`, `urlaubsverwaltung`, `voucher-vault`. Group 2 — the charts whose behaviour hangs off an nginx annotation — followed: `calibre-web-automated`, `outline`, `paperless-ngx`, `zipline` (all four `proxy-body-size`) and `freesailarr` (`rewrite-target`). **Group 3 — the special shapes — finishes the sweep**: `homeassistant`, `keycloak`, `bulwark`, `matrix-synapse`, `stalwart`, `ytdlp-web-player`. That leaves exactly one chart, `rss-bridge`, which is frozen for changes (#51) and is why **#114 cannot be closed while #51 is open**.

  **Three of #114's descriptions of the group-3 shapes did not survive contact with the charts**, and the corrected list is worth having here because it is the one somebody will copy from:

  | #114 said | Measured |
  |---|---|
  | `keycloak` and `bulwark` have **two** Ingresses each | `keycloak` has two (public and admin hostname) and gets two routes. **`bulwark` has one.** |
  | `bulwark` is the second `rewrite-target` chart | It has none and cannot have one — `NEXT_PUBLIC_BASE_PATH` is compiled into its bundle, so it cannot be served under a sub-path and its route is one rule at `/`. The grep that produced the claim hit the chart's *refutations* of the idea. |
  | `matrix-synapse` renders its Ingresses **in a loop over several apps** | That is `freesailarr`. `matrix-synapse` renders **one** Ingress object with **two host rules** — the homeserver at `/` and the delegation nginx at `/.well-known/matrix` — which has to become **two** `HTTPRoute` objects, because `spec.hostnames` applies to the whole object and these two want different paths and different backends. |
  | `stalwart` renders **per node group** | Its Services and StatefulSets do; the routing object does not and must not. One Ingress targets exactly `ingress.nodeGroup`'s own http Service, and the route keeps both of that template's guards — the management surface is not safe to spread over a group that may not run the management tasks. |

  Three further group-3 specifics: **`keycloak`'s admin Ingress did not hang off `ingress.enabled`** before the sweep (`ingress.admin.enabled` gates it, and that value is not an Ingress switch — it also sets `KC_HOSTNAME_ADMIN`), so `ingress.enabled: false` did not mean "no Ingress" and the object would have gone on claiming the admin host next to the admin route; it is gated on both now, which leaves the default render unchanged and only alters a combination that could not be coherent. `httpRoute.admin.parentRefs` falls back to `httpRoute.parentRefs` exactly as `ingress.admin.className` falls back to `ingress.className`, because **"which data path" is a property of the Gateway in this API, not of the route** — an internal Gateway there is what an internal ingress class is in the other mode. And **`stalwart` is the only chart in the sweep that still renders a `Certificate` in Gateway mode**: its mail certificate covers STARTTLS on 25 and implicit TLS on 465/993, which pass through neither an ingress controller nor a Gateway, so `ingress.domain` stays required there and the object must not be removed as part of a migration — deleting it drops SMTP and IMAP back to a self-signed certificate while every Kubernetes object stays green.

  `garage` was one of #114's named special cases and was done in group 1: it ships `ingress.enabled: false`, so both of its modes are off by default and its route is exercised by a render overlay rather than by the install-test, which keeps its Ingress. `homeassistant` is the same case and the same treatment — its `ingress.enabled` default is `false` too, so group 3's overlay for it **adds** coverage the install-test never had rather than trading one object for another. `paperless-ngx` was the one chart in the repository that still rendered its Ingress **unconditionally**; group 2 gave it an `ingress.enabled` switch defaulting to `true`, because without a second value path there is nothing for the mutual-exclusion abort to name and the `required()` on `clusterIssuer` would have kept demanding a value the Gateway mode does not use.

  **One place where the Gateway API is the better fit, found in group 3**: Matrix delegation. The server name is normally an apex domain that already serves a website, and under ingress-nginx two Ingress objects cannot both claim one hostname — the admission webhook rejects the second — which is why `matrix-synapse` has to carry the `/.well-known/matrix` rule *inside* its own single Ingress object. HTTPRoutes attached to the same listener for the same hostname are **merged** by the implementation, longest match winning, so in Gateway mode that route can sit beside whatever owns the apex without either side knowing about the other.
- **Documented exception — freesailarr, and it extends to the Gateway mode**: the chart bundles several independently exposed apps in one pod, so it has **no** `ingress.domain` — and therefore **no `httpRoute.hostnames`** either, the only chart without that key. A chart-global list of hostnames could not say which of the ten apps it belonged to, so the hostnames stay where they already are, in `ingress.host` and `<app>.ingress.host`; both are read by both modes and neither is an Ingress switch. In Gateway mode the per-app shape renders one `HTTPRoute` per app (`<release>-<chart>-<app>-httproute`) and the shared shape exactly one (`<release>-<chart>-httproute`) with one rule per app — qBittorrent included, since its prefix-stripping rewrite is a per-rule filter there, and it is the one app contributing **two** rules: the rewrite plus a 301 from the bare, slashless prefix to the slashed form (Gitea #118). Without that redirect the slashless path is stripped to `/`, answered 200, and the page's relative links resolve to the root of the shared host — a different app. **The Ingress mode does not render it and cannot**: nginx offers only a `configuration-snippet` annotation, which is ignored unless the controller runs with `allow-snippet-annotations=true` (off by default since ingress-nginx 1.9, and it means accepting annotation risk level `Critical`), so the chart documents the snippet in `values.yaml` instead of shipping a no-op. It is the one place where the Gateway mode closes a gap rather than merely replacing machinery. It offers both shapes. By default each app carries its own `<app>.ingress.enabled` plus a required `<app>.ingress.host` (required only while that app's ingress is enabled and it is not on the shared domain) and renders its own Ingress object `<release>-<chart>-<app>-ingress` with its own TLS secret `tls-<release>-<chart>-<app>-ingress`. Setting the global `ingress.host` switches every app without a host of its own to the shared-domain shape under `<app>.ingress.path`; qBittorrent (no base path, needs a prefix-stripping rewrite) gets the extra object described above, and seerr as well as mediathekarr are pinned to `/` — mediathekarr because its UI links assets with root-absolute paths, which a rewrite cannot fix; with both of them on one shared host the duplicate-path guard fires and one of them needs a host of its own (its ingress is therefore off by default). `clusterIssuer`, `className` and `annotations` stay global defaults under `ingress.` and are overridable per app.

### 5. Security hardening

- Defaults on every workload: `automountServiceAccountToken: false`, non-root user, `readOnlyRootFilesystem: true`, all capabilities dropped, `allowPrivilegeEscalation: false`.
- Pod and container contexts are values-configurable via `securityContext.pod` / `securityContext.container`.
- Writable paths (e.g. `/tmp`, nginx cache dirs, s6-overlay's `/run`) are emptyDirs so the root filesystem stays read-only.
- Where an image cannot satisfy the full standard (e.g. Home Assistant must start as root for s6-overlay, paperless needs a root init container to hand over `/run`, samba needs its capabilities), the deviation is minimal and **must** be justified with a comment in that chart's `values.yaml`.

### 6. Probes

- Every workload has liveness, readiness and startup probes (documented exception: the paperless-ngx consume CronJob).
- Probes are values-configurable with calibrated defaults (the startup probe gates liveness/readiness; its budget is documented in a values comment per chart).

### 7. Resources (auto-limits)

- Resource requests ship as chart defaults in `values.yaml` and are user-overridable.
- Limits are computed by a per-chart `_helpers.tpl` helper (reference implementation in `template-chart/templates/_helpers.tpl`): an explicitly set limit always wins, missing memory and ephemeral-storage limits default to `resources.limitFactor` × the request (factor default `3`), and a CPU limit is only rendered when set explicitly — otherwise CPU stays unlimited (it is compressible, throttling instead of OOM-killing). The helper is copied into every chart rather than shared through a library chart — which would be a second artifact to publish, version and pull for a helper that changes about once a year; the resource-quantity render test above exists because a copy can carry an old bug into a diff nobody reads.

### 8. Secrets

- Secrets are **1Password `OnePasswordItem` CRs**: values expose `secrets.<name>SecretRef` holding an `op://` item path; the 1Password operator materializes the Kubernetes Secret in-cluster.

### 9. Config changes must restart the workload (#82)

- **Every workload that consumes a ConfigMap carries a `checksum/config` annotation on its pod template** — one line per ConfigMap it consumes:

  ```yaml
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/<chart>.configmap.yaml") . | sha256sum }}
  ```

  Reference implementation: `template-chart/templates/xxx.deployment.yaml` and `xxx.sfs.yaml`. Where a workload consumes several ConfigMaps, use one annotation per file with a distinguishing suffix (`checksum/config-<name>`).
- **Why it is mandatory, not optional**: without it a changed ConfigMap does not change the pod template, so nothing restarts and the running container keeps the old config — silently. `subPath` mounts (the usual form here) are never refreshed by the kubelet at all, and most apps read their config only at start. In #82 this produced a real outage: the router kept routing to Services that no longer existed while every bundle was green and the pod was Ready.
- **The annotation is content-based, not time-based**: the hash is taken over the rendered ConfigMap, so an unchanged config renders an identical hash and `helm upgrade` is a no-op. It never degenerates into a restart-on-every-upgrade.
- Adding the annotation to an existing chart is a **major** bump (§2): runtime behavior changes — a config change now costs a restart. Name that cost in the PR (some workloads take minutes to come back).
- **Secrets from `OnePasswordItem` cannot use this mechanism.** Helm never sees their content — the 1Password operator fills the Secret in-cluster after rendering, so there is nothing to hash. A chart that mounts such a Secret as a file documents the limitation in its `values.yaml` where it cannot close the gap.
- **What *can* be covered is the secret's `op://` path**, via a **separately named** `checksum/secret-ref` annotation over the rendered `OnePasswordItem`:

  ```yaml
  checksum/secret-ref: {{ include (print $.Template.BasePath "/<chart>.secret.yaml") . | sha256sum }}
  ```

  Repointing `secrets.<name>SecretRef` at a different vault item then takes effect instead of silently leaving the old credentials in the running pod. The separate name is deliberate: `checksum/config` promises "the config changed", `checksum/secret-ref` only "the config *source* changed" — merging them into one annotation would overstate the weaker guarantee. Reference implementations: `minecraft-server`, `homeassistant`.
- **`operator.1password.io/auto-restart` only works for Deployments — do not set it on a StatefulSet.** Verified against onepassword-operator **1.12.0** (`pkg/onepassword/secret_update_handler.go`): `restartWorkloadsWithUpdatedSecrets` enumerates exactly one workload type, `appsv1.DeploymentList`, and `getPodTemplate` accepts only `*appsv1.Deployment`. The RBAC role grants `statefulsets`, and v1.11.0 generalized the restart code (`restartWorkload`, pod-template stamping via `operator.1password.io/last-restarted`), but the PR that would add StatefulSets is still open — so the annotation on a StatefulSet is a silent no-op. Setting it there is **worse than not setting it**: it advertises a guarantee that does not exist. Affected charts (`matrix-synapse`, `samba`, `homeassistant`) instead document the manual `kubectl rollout restart statefulset <release>-<chart>-sfs`. Re-check this when the operator version is bumped; a Deployment-based chart may set the annotation.
- If a Deployment-based chart ever does set it, mind the precedence: the annotation on the **`OnePasswordItem`** is copied onto the Kubernetes Secret and read first, so a `false` there **overrides** a `true` on the workload — the opposite of what the operator's usage guide suggests. Lookup order is Secret (from the `OnePasswordItem`) → workload → namespace → the operator-global `AUTO_RESTART` env var. The trigger is the 1Password item's **version counter**, not its content, so a re-save with identical values still restarts.

### 10. Environment variables

- Every workload container offers an additive `env` value block, appended to the env entries the chart always sets.
- Entries are rendered through `tpl` (so values may contain Helm templating) and support the `value`, `secretKeyRef` and `configMapKeyRef` forms (pattern: `template-chart/templates/xxx.deployment.yaml`).
- **Additive means additive: an entry repeating a name the chart already sets does not replace it, and what the pair does depends on when it arrives.** At install both entries reach the pod and the later one — the user's — wins. On an *existing* release the same entry is lost: Kubernetes merges `env` by name, the pair collapses to the chart's own value, and `helm upgrade` still reports success. Measured on `urlaubsverwaltung` (helm 3.19.0, Kubernetes 1.35): `helm get manifest` shows the variable twice while the live Deployment shows it once, carrying the chart's value. The failure mode is silence, so a chart whose `NOTES.txt` tells operators to override one of its own variables has to say that the override only takes at install time — and that a green upgrade proves nothing, the value belongs read back off the Deployment.

### 11. Scaling

- Every Deployment/StatefulSet has a `replicas` value (default `1`, the values path per workload matches its section) that also accepts an explicit `0` for per-workload fine control.
- Every chart has a global `scaleDown` flag (default `false`): `scaleDown: true` takes precedence over all `replicas` values, renders every Deployment/StatefulSet with `replicas: 0` and suspends CronJobs. It **stops** the app but does not uninstall it — PVCs, Secrets, Services and Ingress stay in place; setting it back to `false` brings the workloads back up.
- Implemented via a per-chart `<chart>.replicas` helper in `_helpers.tpl` (reference in `template-chart`); the helper uses an explicit nil check (`kindIs "invalid"`) because Helm's `default` would swallow an explicit `0`.
- **A chart whose default is more than one replica ships a `PodDisruptionBudget`** — `policy/v1`, named `<release>-<chart>-pdb`, `minAvailable` as a value (default `1`), selector matching the workload's `app` label. `keycloak` (Gitea #83) is the first and sets the convention. The reason it is a rule and not a per-chart taste: a node drain is the most common reason a pod moves — it is what a node upgrade does — and without a budget the drain is free to take every replica at once, so the second replica buys nothing against the failure it was added for.
- **And it is rendered only while the effective replica count is `>= 2`.** `minAvailable: 1` with exactly one replica forbids every voluntary eviction, so `kubectl drain` blocks forever on a pod nothing may move — and the message names the PDB, not the replica count, which sends the reader to the wrong file. The same applies to `scaleDown: true`, where there are no pods and the object would describe a promise about nothing. So the condition is on the **computed** replica count (the `<chart>.replicas` helper), not on the raw value, and `NOTES.txt` prints which of the two happened rather than leaving it to be inferred from `kubectl get pdb`.
- **A chart whose replicas are a property of a *group* gets one PDB per group, with that group's own count** (Gitea #96; `stalwart` is the first, one node group per cluster role). The condition above then applies per group — with `full: 2` and `mx: 1` exactly one object is rendered — and `NOTES.txt` lists the outcome per group rather than once for the release. The name carries the group (`<release>-<chart>-<group>-pdb`), and so does the selector.
- **A budget without a distribution rule is half an answer.** The PDB keeps a drain from taking every replica at once; it says nothing about where the replicas were put in the first place. Where a chart's replicas are meant to survive the loss of a node, it also offers a **per-group switch that renders the `topologySpreadConstraints` entry itself, with a selector the chart fills from its own pod labels** — never one the operator writes, because a spread constraint whose selector matches no pod is trivially satisfied and distributes nothing, with a green rollout and no message. `stalwart` sets the convention: `nodes.<group>.spreadAcrossNodes` (default `true`, gated on the same computed `>= 2`), `maxSkew: 1` on `kubernetes.io/hostname`, `whenUnsatisfiable: DoNotSchedule` so the distribution is a promise rather than an effort, and a generic per-group pass-through (`affinity`, `nodeSelector`, `tolerations`, `topologySpreadConstraints`) for everything the switch does not cover. Three traps come with it and all three belong in the chart's `values.yaml`: `nodeAffinityPolicy: Ignore` must be set **explicitly** (its API default is `Honor`, under which a narrowing `nodeSelector` shrinks the set of counted domains until both replicas fit in one, giving skew 0 — co-location with no `Pending`, no event, and a selector that is *correct*, so no CI check can see it); that field is **beta** and is pruned without a word below Kubernetes 1.26, which brings that exact gap back; and the replica ceiling is **a rule, not a number** — as many replicas per group as there are schedulable workers, because `nodeTaintsPolicy` defaults to `Ignore` and control-plane domains therefore hold the minimum at 0 permanently. That ceiling comes from `nodeTaintsPolicy` and is **not** a consequence of setting `nodeAffinityPolicy`; the two are documented separately for that reason. A user entry colliding with the chart's own on `(topologyKey, whenUnsatisfiable)` **aborts the render naming both values**, because the two alternatives are an API error pointing at a list index and a silent merge over `patchMergeKey: topologyKey` that drops one selector and goes green.
- **Both selectors are held against the rendered pod labels on every PR** by `ci/run-selector-match-test.sh` — see "Selector-match render test" above. Needed because the shared-helper rule above is a convention, and a convention is the thing a copy breaks.

### 12. Values that must not lie (#68, #93, #99, #215)

Every chart ships a **`values.schema.json`** with `additionalProperties: false` on **every chart-owned object level** — not just the top one. The gap this closes is a key that looks like configuration and does nothing: the pilot (#68) found three of them in one chart, one of them nested three levels deep and quietly leaving a game world unpinned. Objects that are passed **verbatim** into the pod spec stay open (probes, `securityContext.pod`/`.container`, `global`, free-form annotation maps): Kubernetes owns their schema, and restricting them would only reject fields that do not exist yet. The schema **adds to** `required()`, it does not replace it.

Reference implementation: `template-chart/values.schema.json` and `template-chart/templates/_helpers.tpl`.

**1. A value read outside its obvious template needs its own guard.** Before putting an Ingress behind `ingress.enabled`, check whether anything else reads `ingress.domain` — **in `templates/` and in `values.yaml`**, because `env` entries are rendered through `tpl` and hide such reads. If something does, that read needs its own `required()`; the gate then frees `clusterIssuer` but *not* `domain`. This happened in six charts, and matrix-synapse shows why it matters more than the count: its delegation nginx serves `ingress.domain` in the `/.well-known/matrix/{server,client}` endpoints through which **federation finds the homeserver**. Gating without the guard publishes

```
{"m.server": ":443"}     {"m.homeserver": {"base_url": "https://"}}
```

— rendered cleanly, pod green, federation broken. Check per chart and in both directions: in `calibre-web-automated` nothing outside the Ingress reads the domain, and there the gate frees both values.

**2. Probes, security contexts and resources go through the null-safe helper, never behind an `if`.** An override written without children is YAML null and erases the block: the container renders `securityContext: null` (unhardened, healthy, silent), a probe disappears, or `resources` loses its requests and computed limits. The `defaultedDict` helper defines the opposite semantics — an empty block means "chart defaults apply" — while an explicitly set value still wins, **including `false` and `0`** (hence the `kindIs "invalid"` check; `default` and `mergeOverwrite` both swallow zero values). Do **not** wrap these blocks in `{{- if }}`: a guard does not render a visible `null`, it removes the probe entirely, which is the same bug with no evidence left.

  The rule is about the **shape, not about those three block kinds**. `satisfactory` shipped 7.0.0 with `game` null-safe and its two neighbours not (Gitea #16): `--set server=null` aborted the render, and a single erased key was the quieter half — a nulled `server.gamePort` rendered `port:` and `nodePort:` empty into the Service and an empty `containerPort` into the StatefulSet, rejected by nothing until the API server saw it. Every chart-owned block a template dereferences belongs behind the helper, and a chart where only *some* of them do is worse than one where none do: whoever sees `game=null` fall back cleanly assumes `server=null` does too.

  **A deliberate exception is fine; an undocumented one is the bug.** `satisfactory.storage` is left un-defaulted on purpose — an erased size yields an invalid PVC that fails at apply time rather than a volume quietly sized by the chart — so `server: null` still renders a PVC without a size. That is a defensible trade and it is *not* the null-safety bug next to it; what makes it safe to leave standing is that the schema, `values.yaml` and the chart page all name it as an exception and point at the issue where it is argued (Gitea #39). Note also where the trade actually bites: only on a **new** install. For an existing release a fallback to the default size would fail too, because a PVC cannot shrink. Do not overturn a reasoned exception inside a bugfix patch — the patch then stops being one.

  **And `helm template` does not render `NOTES.txt`.** A value read there — `server` is read in four files in that chart — stays broken after a fix that looks green in every render, until someone runs a real `helm install`. `helm install --dry-run=client` is not the way around it either: it still wants a reachable API server (measured on helm v3.22.0). What works without a cluster: in a throwaway copy of the chart, wrap the verbatim `NOTES.txt` body in a `{{- define }}` and `include` it into a scratch ConfigMap, which `helm template` *does* render — same engine, same root context, only the sink differs.

**3. Do not hang liveness on an external dependency where a dependency-free path exists.** `zipline` deliberately probes `/` for liveness and `/api/healthcheck` only for readiness and startup, so a database outage cannot restart the pod. Where no such path exists — `outline` returns 500 on `/` when the database is down — all three probes may share the endpoint, but the reason belongs in a comment on the helper, or the next reader will "fix" the inconsistency.

  The other half of the same rule: readiness **must** hang on the dependency, and a separate path is not proof that it does. `urlaubsverwaltung` probed `/actuator/health/liveness` and `/actuator/health/readiness` for two years and still stayed 1/1 Ready through a total database outage (#218), because Spring Boot's `readiness` group contains `readinessState` **alone** — the split paths were right, the group content was empty of anything external. Whatever aggregates behind a probe path (Spring health groups, a `/healthz` a framework assembles for you) has to be read off the **running** endpoint, not off the path name; the chart names the members explicitly (`MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,db`) so the composition is in the template rather than in an upstream default.

**4. A fallback reproduces *this chart's* defaults, not the repo standard.** Several charts deviate from the hardening baseline on purpose (no count here — the previous wording named one and went stale the next time a chart landed). A "cleaned up" standard default is not tidying, it is an outage: `calibre-web-automated` must keep `readOnlyRootFilesystem: false`, because in read-only mode the image disables its PUID/PGID remapping and the runtime uid silently moves from 1000 to 911 — **making existing volume data inaccessible**. The same applies to non-standard values that look like mistakes: `satisfactory` keeps `limitFactor: 2` (that is what holds the memory limit at the published 8Gi) and `fsGroupChangePolicy: OnRootMismatch` (which avoids a recursive chown of multi-GB game files on every start).

  **The sharpest case is a capability that is granted and never used.** `stalwart`'s fallback has to carry `capabilities.add: [NET_BIND_SERVICE]`, because the image sets `setcap cap_net_bind_service=+ep` on its binary — which makes it a capability-dumb binary, and `capabilities(7)` then has the kernel fail the **`execve` itself** when the bounding set does not carry the capability: *"If the process did not obtain the full set of file permitted capabilities, then `execve(2)` fails with the error `EPERM`."* So `drop: [ALL]` with the standard empty `add` does not produce an unhardened container, it produces `exec /usr/local/bin/stalwart: operation not permitted` and a chart that cannot start. The capability is then spent entirely on the exec check and is never used to bind anything, which is exactly what makes it a magnet for a hardening pass looking for dead grants. This is why a deviation is documented with its **mechanism** and not only its name: "needed for binding" would have been wrong and would have invited the removal.

**5. A probe default carries the probe's context, not just its timings.** Two different symptoms, both from real charts: `teamspeak-server` addresses its TCP port by the name `file-transfer` because the voice service is UDP and cannot be probed — a default without that name leaves the workload **with no working probe at all**. `patchmon` sends a `Host` header because the server answers **403** to any request whose Host is neither loopback nor a CORS_ORIGIN host — a default without it means the pod never becomes Ready. Port names, `Host` headers and `exec` commands belong in the fallback; where the context is dynamic, pass the root context in and rebuild it (patchmon derives the header from `ingress.domain`).

**6. A value that reads as the negation of the variable it feeds must actually be negated — and `not` cannot do it.** `satisfactory` shipped `game.seasonalEvents` straight into the image's `DISABLESEASONALEVENTS` (#215): the default `true` switched the events **off** in every install for the chart's whole life, while the schema described the same key as "true means events are enabled". The fix is the loud one — rename the key so it says what it does (`game.enableSeasonalEvents`) rather than silently inverting the template under existing releases, because the old key then fails schema validation instead of quietly meaning the opposite. Two mechanics come with it: Go's `not` treats **every non-empty string as true**, so `not "false"` is `false` and the string form these schemas deliberately accept flips the wrong way — compare explicitly (`eq (lower (printf "%v" $v)) "true"`) instead. And a value that cannot be read as either boolean belongs in a `fail`, not in an implicit branch: `enableSeasonalEvents: "yes"` must abort the render, not be quietly filed under "off". When such a rename lands, the new default is the one that **reproduces what the chart already rendered**; changing the meaning and the behaviour in the same release turns the loud fix back into a silent one.

**Two things about the mechanics that are easy to lose:**

- **`null` reaches the schema in two cases out of three, and which one you are in depends on the chart's default, not on what the user wrote.** Measured on Helm 3.22.0 with a probe chart, because the wording here was wrong before and was applied to exactly the case it got wrong:
  - a user `null` over a **non-null** chart default (`--set x=null`, or `x: null` in a values file) is treated as a deletion and the key is removed *before* schema validation — the schema never sees it, and only the helper can catch it;
  - a **`null` chart default** survives into validation with or without a user `null` on top of it, because there is nothing to delete it against, and the schema sees the key present with value `null`;
  - a key **without any default** (e.g. `resources.limits`) likewise survives, and a strict `"type": "object"` aborts the render.

  That is why the helper **and** the widened `["object", "null"]` / `["string", "null"]` types are both needed: each covers a case the other cannot. Everything else stays strictly typed.
- **A value that has no valid default cannot be enforced by `values.schema.json` at all, and `required()` is the whole enforcement.** The chain is short and it has no exit (#18): the key ships as `null`, so it is *present* at validation time, and JSON Schema's `required` tests presence and not null-ness — the array never fires, under any input. The two ways to make it fire both break the rule that `helm lint --strict` passes on chart defaults, since lint validates those defaults against the schema: dropping the `null` default gives `missing property '<key>'`, and narrowing `["string", "null"]` to `"string"` gives `got null, want string` — each a red `validate-charts` run on a chart nobody touched. So such a key is guarded with `required()` in the template and **described** as required in the schema; a `required` array next to it would be a guard that cannot fire, which is the same defect as a value that does nothing, dressed as a safeguard. This is why no chart in this repo carries one.
- **Validate every schema against the real values, not only against chart defaults.** Defaults are valid by construction, so they prove nothing. Render each chart against its bundle in `cluster-config/fleet-cd/` **and** against `helm get values` — the two differ more often than expected (four of ten bundles in one wave, always with the repo file being the worse one). This step caught a schema of mine that would have broken three live paperless releases, because all three write `tikaGotenberg.enabled` as the string `"true"` and the first draft demanded a boolean. Result per chart is a **list of rejected keys**, which is the cluster side's work order before the version is pulled up.

### 13. Everything else

- `icon:` is mandatory in every `Chart.yaml`. **Documented exception — freesailarr, the only chart without one**: the rule everywhere else is "the icon is this chart's one upstream's logo", and a chart bundling ten independently versioned projects has no such upstream — picking one of the ten would imply a "main app" the stack deliberately does not have. That is the whole criterion, and it applies to no other chart in the repo: a single-upstream chart without an icon is a missing icon, not an exception. The omission costs an INFO line in `helm lint --strict` (`icon is recommended`), which still exits 0.
- `helm lint --strict` must pass without findings.
- **Every chart ships a `templates/NOTES.txt`** (#43), `template-chart` included — so this is a requirement for a new chart, not a nice-to-have it can defer. No list of names here on purpose: a count goes stale the moment the next chart lands, which is exactly how the previous wording came to claim a number it never matched. Reference implementation: `template-chart/templates/NOTES.txt`, a placeholder-carrying template rather than finished prose, so a new chart adapts it instead of starting from an empty file. The bar it sets: print what an operator would otherwise have to look up in the templates (the rendered access URL, **which fields the 1Password item must carry**, the options actually active in this release, and the chart's known traps), conditional on the values really set. A `NOTES.txt` that only repeats the values is skimmed once and never read again. **No real domain anywhere** — the URL renders from `ingress.domain`, and where an example is unavoidable it is `example.com`. A `NOTES.txt` renders no Kubernetes object, so writing or reworking one is purely additive: a minor bump.
- CI: every installable chart has fixtures under `ci/<chart>/` (`test-values.yaml`, optionally `fixtures.yaml`/`fixtures.env.yaml`/`settings.env`) or a justified entry in `ci/excluded-charts.txt` — the install-test fails if both are missing.
### 14. Chart dependencies

Until `ytdlp-web-player` (#214) this section read "charts have no dependencies (`charts/` directories are empty)". That rule is gone, replaced by a bar rather than a ban — a dependency is **admissible, not free**, and the reason it stopped being a ban is worth keeping: the chart needed an oauth2-proxy in the data path in front of an application that ships no authentication at all, and proxy and app are only useful wired to each other.

Three charts carry one, and comparing them is the quickest way to see where the bar sits. `ytdlp-web-player` takes **upstream's** `oauth2-proxy` because the app has no authentication and the two are useless apart; there, point 2 above applies in full. `stalwart` (Gitea #65) takes **one of our own published charts**, `valkey`, because Stalwart's **cluster coordinator rides on the in-memory store connection**, so a multi-node install does not work without one. `paperless-ngx` (Gitea #78) takes the same one, because `PAPERLESS_REDIS` is the Celery broker and the application therefore cannot run without a Redis at all — before the subchart, every installation needed a second Helm release whose address lived in a 1Password item and in neither chart. All three are `condition:`-gated and off by default: the subchart is the path for an installation that has no Valkey yet, and one that already has points its connection value at that instead, so the dependency renders nothing.

**The same subchart, opposite configuration — and that comparison is the useful part.** `stalwart` runs it as a **cache** (`persistence.enabled: false` + `allkeys-lru`): its in-memory store holds rate-limiter counters, tokens and pub/sub, all reconstructible, and the cost it accepts is that the limiters fail open under memory pressure. `paperless-ngx` runs it as a **queue** (PVC + `noeviction`, the subchart's defaults): the broker's contents are work, and a dropped consume task is a document that is silently never processed. `allkeys-lru` would not be a trade there but actively wrong, because LRU evicts typeless across the whole keyspace and a backlog of waiting tasks is exactly what it reads as cold. The price `paperless-ngx` accepts instead: the same instance also serves the Django cache and the Celery result backend (both Redis in 3.x), so a full instance **refuses writes** — new tasks included — rather than dropping cold keys, and the answer is memory sizing, not a policy change. The valkey chart treats the two values as a **pair** and warns against mixing them; these two charts are the worked examples of picking either half deliberately.

**A dependency on one of our own charts is cheaper in one way and more dangerous in another.** Point 2 stops applying: there are no third-party defaults to leave unpatched, because the subchart follows these guidelines by being one of ours. In exchange it adds a hazard an external dependency does not have, and it is sharp enough to be a rule:

> **Never change a chart and a chart that depends on it in the same pull request.**

The dependency resolves from the **published index**, not from the working tree, and a chart is published only by a push to `main` that touches it. So a combined PR runs every `helm` command — lint, package, render **and the install-test** — against the **old** subchart, goes green on an artifact that is not the one in the diff, and leaves the subchart's own change exercised by nothing. Two consecutive PRs: change the subchart, merge, let it publish, then raise the `version:` in the dependent chart.

Two mechanical details of a self-referencing dependency, both measured for #65: the index is **anonymously readable**, so `ci/helm-dependencies.sh` needs no credential and resolves the exact build-metadata version (`1.0.0+up9.1.2.alpine`) from the URL alone with an empty `HELM_REPOSITORY_CONFIG` — the state a fresh runner is in. And Renovate's built-in `helmv3` manager does follow a self-hosted Gitea index: a dry run (Renovate 44.132.2) with the dependency pinned low reported `newVersion: "1.0.0+up9.1.2.alpine"`, `versioning: "helm"`, timestamp read from our index. Without that second result the published-version form would have had no update path at all, which is why it was measured rather than assumed.

One more trap, and it is the reason `stalwart` gates its subchart on `valkeySubchart.enabled` rather than the conventional `valkey.enabled`: **everything under the subchart's key is handed to the subchart as its values**, so a `condition:` key placed there is validated against *its* schema. Our charts ship `additionalProperties: false`, so `valkey.enabled` is rejected by the subchart — and because Helm skips a **disabled** subchart's schema validation, the error surfaces only for whoever first switches it on. Put the condition on a key of the parent's own.

**When a dependency earns its place.** All three, not two of three:

1. The dependent piece would otherwise be a second Helm release that only exists to serve this one, with the wiring between them living in neither chart.
2. Upstream is the right owner of it. We consume their chart as it comes: **we do not patch it, do not re-harden it, do not re-implement its values.** Where their defaults fall short of the guidelines above, that is **documented in our `values.yaml`**, not fixed by us — and a values-level fix stays available to whoever installs it.
3. It is **optional and off by default** (`condition:` in `Chart.yaml`), so the chart is complete without it and a default install renders none of its objects.

**What it costs, all of it visible before the next one is added:**

- **The subchart is invisible to everything downstream.** Its version is inside our packaged `.tgz`, so anything that watches Helm release versions — Fleet's `fleet.yaml` in `cluster-config` included — sees only *our* chart version. A new upstream release, security fix included, reaches a cluster **only** when we publish a new version of the outer chart. Nothing reports the gap. Renovate's built-in `helmv3` manager opens the PR (no `customManagers` entry needed — the repository URL and version sit together in one file) and `ci/renovate-bump-chart.sh` bumps the chart patch that ships it; that chain is the entire safety net. State this in a comment at the pinned version, not only in a review.
- **Every pipeline that touches the chart needs a network fetch first.** `helm lint`, `helm package`, `helm template` and `helm install` all exit 1 while a declared dependency is missing from `charts/`, so `ci/helm-dependencies.sh` runs before each of them (see CI/CD above). One of those call sites is the resource-quantity render test, which runs over **every** chart on **every** PR — a dependency-carrying chart can therefore turn a third-party Helm repository's bad day into a red pipeline on a PR that touched nothing related.
- **`charts/*.tgz` and `Chart.lock` are not committed** (`.gitignore` says why): the archive is a third-party binary nobody reviews in a diff, and a committed lock would arrive stale because Renovate rewrites `Chart.yaml` and cannot regenerate it — which is also why the pipelines run `helm dependency update` and not `helm dependency build`.
- **The repo's own mechanisms stop at the subchart boundary.** `checksum/config`, the 1Password auto-restart annotation and the `scaleDown` flag reach our templates only. A chart that renders a `OnePasswordItem` for its subchart documents the manual `kubectl rollout restart` that replaces the missing restart-on-change.

Two implementation notes that cost time otherwise: a values key containing a dash is unreachable as `.Values.oauth2-proxy` (that parses as a subtraction) and needs `index .Values "oauth2-proxy"`, while `condition:` in `Chart.yaml` splits on dots only and needs no escaping. And inside a subchart `.Chart.Name` is the **subchart's** name — a value templated into it (`config.configFile` in this case) must spell the parent chart out literally; only `.Release.Name` is shared.

Charts that need no dependency still have none, and `charts/` stays empty for all of them. `template-chart` remains dependency-free: the scaffold should not carry a `dependencies:` block a new chart has to remember to delete.