helm-charts/README.md

helm-charts

Collection of some self-made helm-charts. Each top-level directory is a standalone Helm chart that gets packaged and uploaded to a private, classic (index-based) Helm registry by CI.

This repo lives on the Gitea instance JLab-Git. Issues, pull requests, CI and Renovate all run there, and main there is the leading copy. github.com is a push mirror with Actions and Renovate switched off. The workflow files under .github/workflows/ are read by Gitea unchanged; "Where this repo lives" below names the places where a Gitea runner behaves differently from a GitHub one, each of them measured rather than assumed.

The knowledge base is the wiki. It holds one page per chart — prerequisites, an installation walkthrough, the complete list of configurable values and the chart's operating traps — plus the long form of the workflow, the pipelines, the style guide and the lessons each rule was bought with. This README stays the overview for someone reading the code; anything an operator needs is in the wiki. Keeping it current is a required step of the workflow (CLAUDE.md), not an afterthought.

A reading note on #n. Issue numbers in this file point at the GitHub tracker. Most of them name closed issues, which were not migrated and exist only there. Gitea resolves a bare #n against its own counter instead — where, for example, #22 is an unrelated throwaway pull request rather than the install-test issue this file means. The seventeen issues that were still open at the move are Gitea #5–#21 and #24; the mapping is a comment on Gitea #14.

Charts

Every chart has a wiki page of its own — https://git.jk-dev.de/jlab-cluster/helm-charts/wiki/<chart> — carrying the installation walkthrough, the required 1Password fields and the full values reference. The table below is the one-line version.

Chart	Deploys	Notes
apache-tika	Apache Tika	Deployment + Service only (no ingress)
bentopdf	BentoPDF	Stateless Deployment + config.json ConfigMap, subPath-mounted into the web root and always rendered (empty renders as {})
calibre-web-automated	Calibre-Web Automated	StatefulSet, extra consume-PVC
cyberchef	CyberChef	Stateless Deployment
freesailarr	gluetun VPN media stack (qbittorrent, prowlarr, flaresolverr, radarr, recyclarr, sonarr, seerr, bazarr, and mediathekarr off by default)	Single-pod StatefulSet: all apps share gluetun's network namespace; config volumes via volumeClaimTemplates, media PVCs as existingClaims (optional per-category subPath, so one claim can serve all three categories). One ClusterIP Service per app (mediathekarr's carries both of its ports); no global ingress.domain. Ingress in two shapes that mix: one host per app (default, each app's host required), or one shared ingress.host with the apps as sub-paths (documented ingress exception below)
gotenberg	Gotenberg	Deployment + Service only (no ingress)
homeassistant	Home Assistant	StatefulSet; the only chart carrying Helm tests (templates/tests/)
matrix-synapse	Matrix Synapse	StatefulSet plus an nginx delegation Deployment/Service
minecraft-router	itzg/mc-router	Deployment + routing ConfigMap + NodePort Service (TCP only, no ingress); mappings is a required value
minecraft-server	itzg/minecraft-server	StatefulSet + PVC + Service + init-script ConfigMap (no ingress); service type is a value (ClusterIP by default, routed via minecraft-router). Only the newest Java line is maintained (chart 5.0.0, #158): the directory carries one line, older ones stay published as frozen artifacts (java8/java17 at 4.0.0, java21 at 4.1.0) and receive no further chart changes — publishing once per line out of one directory is what made the lines drift. To run an older JVM with the current chart, set image.tagOverride; the chart then checks that minecraft.version fits the Java line of the effective tag and fails the render with both values named. Two cases pass unchecked on purpose: a tag without a java<N> part and a minecraft.version that is not a plain numeric x[.y[.z]]. image.tagOverride is deliberately outside Renovate's reach — renovate.json5 tracks appVersion only. Since chart 2.0.0 the tag is an immutable dated upstream release (appVersion: <release>-java8 / -java17 / -java21 / -java25, e.g. 2026.8.2-java8) instead of the continuously rebuilt rolling line tag. Pilot for the strict values.schema.json (#68), which is now binding for every chart
outline	Outline	StatefulSet, separate DB/Redis secrets
paperless-ngx	Paperless-ngx	Largest chart: consume CronJob, SMB-connector PV/PVC, pre/post-consumption scripts ConfigMap
patchmon	PatchMon	One Deployment (the 2.x server serves frontend and /api/* from one process) plus an optional guacd sidecar; OIDC-only sign-in
rallly	Rallly	Deployment, DB/OIDC/SMTP secrets
rss-bridge	RSS-Bridge	Deployment + whitelist ConfigMap; ingress.hosts is a required list of host + optional tlsSecretName, so several hosts can keep their own existing certificates (documented deviation from the one-TLS-secret rule, see its values.yaml)
samba	Samba	StatefulSet, users secret + config ConfigMap
satisfactory	Satisfactory game server	StatefulSet, no ingress
searxng	SearXNG	Stateless Deployment + settings ConfigMap (use_default_settings: true plus a free-form settings.extra passthrough); the signing key is a required OnePasswordItem, an external Valkey an optional second one. ingress.domain is read twice — Ingress host and SEARXNG_BASE_URL — so it stays required with the Ingress switched off
sure	Sure (we-promise, the maintained Maybe-finance fork)	One StatefulSet with two containers: the Puma web process and a Sidekiq worker, sharing one ReadWriteOnce PVC at /rails/storage — that shared volume is the reason they live in one pod. Only the web container migrates (the image entrypoint runs db:prepare solely for the ./bin/rails server command, so the worker is started through args: and skips it). Three 1Password items (app/database/redis); Postgres and Redis stay external. Readiness deliberately probes / rather than Rails' /up, which is a boot check and touches no database; the worker's three probes ask sidekiqmon for this pod's own heartbeat. ingress.domain is read for APP_DOMAIN and the WebAuthn relying party, so it stays required with the Ingress switched off
teamspeak-server	TeamSpeak server	StatefulSet, no ingress
template-chart	—	Scaffold for new charts (placeholder xxx), never published
urlaubsverwaltung	Urlaubsverwaltung	Deployment, DB + OIDC secrets
valkey	Valkey	One instance per application (cache or queue backend), replacing the shared cluster-redis whose single ACL file coupled six applications together. StatefulSet with one replica + ClusterIP Service, no cluster mode, no Ingress and no value to add one. The password arrives as a finished configuration line (requirepass <secret>) in the 1Password field auth-conf, is mounted as a file and included by the config — never in argv, never in the environment; an init container refuses to start the pod unless that line is really there, because the alternative failure is an open server that looks healthy. maxmemory is derived from the computed memory limit and a configured value above it aborts the render. Persistence and eviction are a pair, not two switches (queue = PVC + noeviction, the default; cache = emptyDir + allkeys-lru). First chart carrying a NetworkPolicy (on by default, Gitea #5), and the first whose probes had to be proven capable of failing: valkey-cli exits 0 on an error reply unless -e is given
voucher-vault	VoucherVault	StatefulSet, DB + OIDC secrets
ytdlp-web-player	ytdlp_web_player	Stateless Deployment, video cache and tool cache as emptyDirs (no PVC). The only chart with a dependency: an optional oauth2-proxy subchart, off by default, wired in front of the app because the application ships no authentication at all — see "Chart dependencies" below
zipline	Zipline	StatefulSet, DB secret

Versioning

Chart.yaml uses the scheme version: <chart-semver>+up<app-version> (e.g. 1.4.0+up1.9.2) — see the chart style guidelines for the full rules (suffix normalization, semver bump levels).

• Bump the chart version in every change that touches a chart directory — a push to main publishes the chart.
• App-version updates go through PRs (so the validate and install-test pipelines run). The manually dispatched Change App-Version workflow (rewrites version:/appVersion: and commits directly to main) exists but bypasses both pipelines. For app major jumps, update stepwise: first to the last release of the old major line (own PR, merge, publish), then to the newest major.

CI/CD

• publish-<chart>.yml — one thin workflow per chart; triggers on pushes to main touching <chart>/** (or manually) and calls the reusable workflow below. A push to a chart directory on main publishes that chart, so the chart version must be bumped in the same change.

• publish-helm-chart.yml — not in this repository any more. The reusable workflow lives in workflow-util/pipelines, the shared building-block repo, and the callers here bind it to a tag: uses: workflow-util/pipelines/.gitea/workflows/publish-helm-chart.yml@v3 (Gitea #66). It still does the same thing — helm package, then a single HTTP POST of the packaged .tgz to the private chart registry, which is a classic, index-based Helm registry and not an OCI one, so helm push is not used, there is no helm registry login session, and every request carries its own Basic auth. What changed is who supplies what:

  • registry-owner: ${{ github.repository_owner }} in each caller's with:. The owner used to be derived inside the workflow (#63); it is now a required input with no default, so the expression is evaluated here, in the calling repository, where the value is measured. That is pipelines' house rule — a shared block never guesses an owner segment — and it is why every caller carries that one extra line.
  • secrets: inherit, unchanged, for OP_CONNECT_TOKEN.
  • the repository variable OP_CONNECT_HOST, which replaced the Connect host that used to be hardcoded in our copy. A shared block must not carry one tenant's infrastructure address; the block's first step fails and names the variable when it is unset.

  The registry credentials and the target URL still come from 1Password Connect (item op://GitHub-Actions/jlab-registry-access/ with username, password, charts-url). charts-url holds only the base up to /api/packages and the block composes <charts-url>/<owner>/helm/api/charts, so no registry address appears in this repository — not even the owner path segment. ci/helm-dependencies.sh is still run before helm package, and it is still our script: the block's checkout clones the calling repository, so it runs from this tree.

  The reasons behind the upload step are not repeated here, because they are no longer ours to keep current: why both halves of the composed URL are guarded, why the credentials go in through curl --config - instead of --user, why --fail-with-body is what makes a failed upload fail the job, and why the file name is built rather than globbed, are all on the publish-helm-chart wiki page of that repository. Nothing was dropped in the move — the upload step's shell body went across unchanged except for two error messages that name the owner as an input.

  Note: the registry accepts a re-upload of an already published version (HTTP 201) and replaces the stored artifact, so a republish of an existing version is still silent — see #160. That is also what makes a manual workflow_dispatch of any publish-<chart>.yml a harmless way to prove the shared block works.

• change-app-version.yml — manually dispatched app-version bump; the chart must be listed in its chart-name choice list.

• ci/helm-dependencies.sh <chart-dir> — not a workflow but the step they all share: fetches a chart's declared dependencies into charts/, and returns silently for the charts that declare none (all but one). It runs in five places, because every helm command this repo builds on exits 1 while a declared dependency is missing: before lint and package in validate-charts.yml, before package in the shared publish block (which is why that block names this script as a prerequisite of its callers — see above), and inside ci/run-install-test.sh, ci/run-quantity-render-test.sh and ci/run-trivy-config.sh. See "Chart dependencies" in the style guidelines for what that costs.

• validate-charts.yml — PR pipeline: detects which charts a PR touches and, for exactly those, runs helm lint --strict, helm package, and a chart-version bump check (template-chart is exempt from the version check). PRs without chart changes skip the build matrix. After the build stage, an install-test job (the same changed-charts matrix, minus the charts listed in ci/excluded-charts.txt) installs each changed chart into an ephemeral k3d cluster on the runner and requires every workload to become Ready — see below. Both matrices follow the changed chart directories, so a PR that only changes the harness under ci/ exercises nothing by itself; the Trivy scan has a fixed-sample fallback for that case, the install-test has none — bump the affected chart's patch version to pull it into the matrix, or run the all-charts install-test after the merge. The resource-quantity render test is the exception to that rule: it runs over every chart on every PR — see below.

• install-test.yml — reusable workflow (workflow_call) holding the install-test job itself: the runner-label expression, the pinned kubectl with its skew check, envsubst, the containerised-job detection, the per-leg k3d cluster and the freesailarr VPN credentials. Its one input is a JSON array of chart names; its callers decide which charts are tested and whether the job runs at all, never how. Two callers: validate-charts.yml and install-test-all-charts.yml. It is not copied for the second caller — a second copy is a second chance to test differently, unnoticed, the same argument that gives ci/excluded-charts.txt exactly one parser.

  It stays in this repository, and that is a decision, not an oversight (Gitea #66, decision 4). It is reusable in form but not in substance: it is welded to the ci/ fixtures, to the ci/excluded-charts.txt parser and to this repository's chart matrix. A building block that only one repository can use is not shared, it is merely moved somewhere else — and moving it would buy nothing while putting a tag between us and a file we change often. publish-helm-chart.yml was the opposite case: it takes a chart name and a registry owner and knows nothing about this repository, which is exactly why it could go. This note exists so the question is not asked again in half a year.

• install-test-all-charts.yml — manually dispatched all-charts install-test: the same job, with a matrix over every installable chart instead of the changed ones. Optional input for a subset of charts (empty = all). See below for when to run it.

• sonar.yml — SonarQube analysis, a hull like the publish ones: it triggers on pushes to main (no paths filter — the analysis looks at the whole tree) or manually, and calls workflow-util/pipelines/.gitea/workflows/sonar-analysis.yml@v4 with secrets: inherit. The project-specific half is sonar-project.properties in the repo root. It is not a gate and it does not block anything — see below.

Where this repo lives, and what a Gitea runner does differently

Everything below was measured on this instance, not taken from documentation, and it is the reason for every deviation in validate-charts.yml. How the move got here is not repeated — that is Gitea #14.

Reusable workflows with secrets: inherit work, so the 25 thin publish-<chart>.yml hulls run unchanged — including now that the workflow they call lives in another repository (Gitea #66). One quirk worth knowing before debugging the wrong thing: a missing secret does not fail the job, it fails job creation, so the run shows no steps at all.

Runner labels mean different machines than on GitHub. ubuntu-latest here has no Docker socket; ubuntu-docker has one. Exactly one job cares — the install-test, because k3d is a Docker client — and its runs-on is therefore an expression rather than a fixed label, so the same file still works on a GitHub runner where ubuntu-docker does not exist. Every other job (lint/package, the quantity render, the Trivy scan) needs no socket and stays on ubuntu-latest.

A matrix runs concurrently inside ONE job, under ONE status context. This is the single most consequential difference and it has two halves. Operationally: all legs share one Docker daemon, which is what makes a per-leg resource name mandatory (see the cluster name below). For branch protection: a required check per chart is not possible, because there is only one context for the whole matrix — and its name carries the unexpanded matrix expression, e.g. Validate Helm Charts / Install-test ${{ fromJSON(needs.detect-changed-charts.outputs.install-charts) }} (pull_request). Required-check names have to be read off a real run; they cannot be copied from GitHub. That example name is from before the install-test job moved into the reusable install-test.yml (Gitea #52) — the matrix expression now reads ${{ fromJSON(inputs.charts) }} and the context may carry the calling job's name as well, so read the current name off the next chart PR before using it anywhere. What is required on main today is validate-charts, as the single context it reports under, alongside a review: per chart is the part that is impossible, not required checks as such. Both are what makes Renovate's platformAutomerge: true safe — see "PR policy".

Expected runtimes, so that a later deviation is visible as one:

	measured here	GitHub, same commit
Create k3d cluster	19–25 s	22–23 s
install-test job, two charts in the matrix (cyberchef + freesailarr)	210–216 s over three runs, spread 6 s	~145 s wall, two parallel VMs
install-test job, freesailarr alone, ten containers and a real tunnel	225 s (measured on the MediathekArr change, Gitea #20)	not comparable — Actions there were already off
install-test job, 20 charts in the matrix — the accidental all-charts run of Gitea PR #48	12m41s, one job, all legs concurrently on one Docker daemon; 19 of 20 charts Ready, the twentieth is Gitea #51	never ran there

The runner is not slower — cluster creation is a touch faster. The difference in total time is structural, not performance: two matrix legs share one runner here, while GitHub gives each leg its own VM. Reading it as "Gitea is slower" leads to the wrong conclusion.

aquasecurity/setup-trivy cannot be used. It runs actions/checkout against aquasecurity/trivy on github.com and hands it github.token — which here is a Gitea token. github.com answers 401 and checkout ends on fatal: could not read Username for 'https://github.com', naming neither the action nor the token. Its actions/cache step is inert here as well. The workflow downloads the release tarball instead: one request, same behaviour on either forge.

Tools are installed, not inherited — and the two cases differ on purpose. kubectl is pinned to match K3S_IMAGE and guarded against version skew, because a client's distance to the API server changes what the test means, and a version that arrives with the runner image drifts with every image rebuild. envsubst is only checked for and fetched if absent, because it substitutes text and the release doing so does not change the result; pinning it would be upkeep without value. That GitHub's image happens to ship both does not make either unnecessary — it only makes the drift invisible there.

Containerisation is detected, not inferred from the platform. k3d publishes the API server port on the Docker host; when the job is itself a container — as every Gitea Actions job is — that host is the dind container, so the https://0.0.0.0:<port> k3d writes into the kubeconfig points at the job container, where nothing listens. The workflow asks the daemon about its own container instead of asking which forge it is on: a self-hosted GitHub runner in a container has the identical problem, and a condition that says "GitHub" while meaning "not containerised" is the kind of lie this repo exists to remove.

Still unmeasured, because they never stood in the way: /dev/fuse and user namespaces. Said as unmeasured rather than as fine.

What a GitHub runner had been providing quietly

Getting the install-test to run here surfaced six assumptions that had never been written down anywhere. Five of them say something about the runner, one about our code, and the split is the point.

The five environment ones are the harmless kind: they fail loudly, immediately, and are fixed on the node side (cluster-config), not here — Docker bridges built at MTU 1500 against a pod network of 1450 (small responses fit in one packet, a 17 MB tarball does not, which is why it looked like a blocked host); fs.inotify limits too low for k3s, so containerd never started and the node never registered; no kubectl and no envsubst in the image; and the kubeconfig pointing at 0.0.0.0.

The sixth is different, and it is the one to remember.

The k3d cluster name was hard-coded as install-test — and that was wrong on GitHub too. It simply could never show there: each matrix leg gets its own VM with its own Docker daemon, so two legs never saw each other's cluster. Here, where all legs share one daemon, it collides immediately — one leg logs Re-using existing network, the other dies on The container name /k3d-install-test-server-0 is already in use, and the first leg's rollback then deletes the shared network out from under the second. Two red legs, and neither message names the cause.

For two years the roomier environment carried that bug instead of showing it. A generous environment is no friend when you want to know whether something is correct.

Two rules for every check added here

Both were learned by getting them wrong in this repo, and both apply to any new verification step, not just to clusters.

1. A check that a broken system passes is worse than no check. kubectl get nodes exits 0 on an empty list, so a readiness check written that way went green against a cluster whose node had never registered — and the failure then surfaced five minutes later, somewhere unrelated. kubectl wait --for=condition=Ready nodes --all is not the shorter way to write it: with zero nodes the condition is vacuously satisfied. Assert the thing you mean (at least one node, and Ready), not that the command answered.
2. Green does not mean checked. A passing job earns exactly as much trust as the assertions inside it. The open case of this kind is Gitea #24: the images in ci/<chart>/fixtures.yaml are not followed by Renovate, so the install-test can keep passing against fixture infrastructure that is years stale. Not solved — named here so it is not mistaken for solved.

Resource-quantity render test (all charts, every PR)

ci/run-quantity-render-test.sh [chart …] (no argument = all charts) renders every chart twice with the fixtures the install-test and the Trivy scan already use, injecting a bare integer memory request at every resources block the chart's values.schema.json declares:

Case	Injected	Expected limit	Catches
bare-int	requests.memory: 1048576, limitFactor: 2	2097152	render aborts
fraction	requests.memory: 999999, limitFactor: 1.5	1499998.5	quantity in exponential notation

The two cases exist because the bug behind #157 — quantities formatted with printf "%v", where the float64 the null-safe fallback helpers produce flips to exponential notation from about 1e6 — has a loud and a silent half. Loud: the reformatted quantity carries the suffix e+06, the helper rejects it, helm template exits non-zero. Silent: the product of a fractional limitFactor and a bare number is reformatted after the suffix check, so the render succeeds and the manifest carries memory: "1.4999985e+06" — rejected by nothing until the API server sees it at apply time. A check on the exit code alone therefore proves only half the case; the second case searches the rendered text for a resource quantity in exponential notation. 999999 stays below 1e6 on purpose, so the fraction case isolates the second formatting site instead of tripping over the first.

Why all charts rather than the changed ones: the resources helper is copied into every chart's templates/_helpers.tpl rather than shared through a library chart. The regression arrives by copy — a new chart started from an older chart, or from template-chart before the fix — and in that PR the copied bug is in no diff anyone reads. Running over everything also removes the list nobody would maintain: adding a chart enrolls it automatically. Two renders per chart, no cluster, a few seconds for the whole repo; the job depends on nothing else and finishes well inside the build-chart runtime.

Two details worth knowing before touching it:

• The resources paths come from values.schema.json, not values.yaml — it is JSON, so python3 reads it without pyyaml (same runner constraint as ci/run-trivy-config.sh), and every chart-owned key must appear there anyway. A block counts when it is #/definitions/resources or carries limitFactor/requests itself, which is what keeps minecraft-server's JVM-sizing minecraft.resources out. A chart whose templates render a resources block while its schema declares none fails the test with exactly that message: the schema is then out of sync with the chart.
• A chart that never reaches the helper is reported, not counted as passing. gotenberg sets every computable limit explicitly in its fallback defaults, and since a null never deletes a default there, no override can free them — nothing is multiplied, so nothing is proven. Those charts appear under "helper not reached" in the job summary rather than as a green tick for a code path that was never entered.

Install-test (PR pipeline stage 2)

For every chart a PR changes, ci/run-install-test.sh <chart> runs against a throwaway k3d cluster: it applies the stub OnePasswordItem CRD plus the chart's fixtures, helm installs the chart with its CI values, and waits for all Deployments/StatefulSets to roll out (per-chart timeout). Since probes and security hardening are active on all workloads, "Ready" means the app actually answers. On failure the job dumps pod descriptions, events and container logs.

The job itself is the reusable workflow install-test.yml; validate-charts.yml calls it with the changed charts, install-test-all-charts.yml with all of them. Everything about how a chart is installed is in that one file, deliberately — see "All-charts install-test" below.

Fixtures live at the repo root under ci/ (deliberately not inside the chart directories, so they are neither packaged into the chart tgz nor trigger the publish workflows on merge):

• ci/common/ — fixtures applied for every chart (schema-less stub CRD for OnePasswordItem)
• ci/<chart>/test-values.yaml — values for the CI install (required for every installable chart)
• ci/<chart>/fixtures.yaml — dummy secrets with in-cluster connection data, throwaway infrastructure (postgres/redis/dex), static hostPath PVs/StorageClasses where the chart pins a storage class or needs RWX, and seed jobs (e.g. synapse generate)
• ci/<chart>/fixtures.env.yaml — fixtures that need a value which must not be committed. The file holds ${VARIABLE} placeholders and nothing else; ci/run-install-test.sh renders it with envsubst and pipes the result straight into kubectl (never printed). Substitution is restricted to the names from REQUIRED_ENV, so an undeclared variable is never filled in and no unrelated $… in the manifest is touched
• ci/<chart>/settings.env — optional overrides: INSTALL_TIMEOUT for slow starters, and REQUIRED_ENV="VAR_A VAR_B" to declare the variables fixtures.env.yaml needs. A missing or empty declared variable fails the install-test loudly, naming the variable — it never skips the chart. Skipping would restore exactly the green-but-meaningless signal this mechanism exists to remove; the one legitimate case of absent secrets, a PR from a fork, belongs in an if on the workflow job rather than in the script
• ci/excluded-charts.txt — charts that cannot run in CI, one per line as <chart> <justification>. Currently: satisfactory (multi-GB image + 10GB+ Steam download), template-chart (scaffold with placeholder image). The justification is mandatory: a line carrying only a chart name is deliberately not an exclusion, because a bare name would drop a chart out of CI with nothing to review. An excluded chart gets no install-test job at all — see below

freesailarr is the only chart using fixtures.env.yaml today: gluetun needs a real Wireguard key, or it never completes a handshake and no app container behind its kill switch becomes Ready. The install-test job loads the key from 1Password Connect for that chart only, using the same pattern as the shared publish block — steps.<id>.outputs.* into the step's own env:, quoted, with :? guards that name the missing op:// field before the cluster is even created. The key in that item is a CI-only Mullvad key, deliberately not the production one: Mullvad ties an address to a key and routes the tunnel to the endpoint it saw last, so a shared key would make every PR run pull the tunnel out from under the running stack — with the symptom in the cluster and the cause in a pull request.

An excluded chart is dropped from the matrix, not skipped inside its job (#231). It used to get a job that ran the script, printed SKIP and exited 0 — and GitHub knows only the exit code, so the check list showed Install-test satisfactory ✓ for a chart whose image was never pulled, indistinguishable from one that really came up in the cluster. detect-changed-charts now removes the excluded charts before the matrix is built, so no job exists for them and nothing claims they passed. Because an absence on its own says nothing, that job writes every excluded chart of the PR with its documented justification into the job summary — a Skipped would only say "not run", and why is the question a review actually asks. The list is parsed in exactly one place: ci/run-install-test.sh --list-excluded emits <chart><TAB><reason> and the workflow calls it instead of re-implementing the format in YAML, since a second parser would be a second chance to select different charts — silently, and in either direction. The script keeps its own SKIP branch for local runs. What cannot be done instead: deciding per chart in an if on the install-test job, to get a visible Skipped entry rather than an absent one. GitHub rejects it with context "matrix" is not allowed here — jobs.<id>.if is evaluated before the matrix is expanded, so the matrix context does not exist there, and the whole workflow ends in failure with zero jobs created (measured on the closed PR #233).

Fixture images avoid Docker Hub where possible (public.ecr.aws/ghcr.io) to dodge runner rate limits. The job blocks the PR on failure — the stabilization phase of #22 ended when Renovate started automerging patch/minor updates, since a chart that no longer starts must not reach main and publish unattended. When adding a new chart, add its ci/<chart>/ fixtures (or an entry in ci/excluded-charts.txt) — the install-test fails if both are missing. The script also runs locally against any existing cluster: k3d cluster create t && ci/run-install-test.sh <chart> && k3d cluster delete t.

All-charts install-test (on demand)

install-test-all-charts.yml runs the same job as the PR pipeline — the same install-test.yml, the same pinned kubectl, the same fixtures — with a matrix over every installable chart instead of the charts a PR changed. workflow_dispatch only, no schedule: a daily run would spend compute on a question that changes rarely, and a red run nobody asked for is a red run nobody reads.

It answers a different question from the PR pipeline. The PR install-test guards against what a change breaks; this one asks whether the environment under an unchanged chart still holds. The install-test matrix follows the charts a PR touches, so a chart nobody has edited since the forge move carries a green history from a different runner — and that green says nothing about today. Run it after a change to the runner, to K3S_IMAGE, to the pinned KUBECTL_VERSION, to the fixtures under ci/ — and after a forge move. It is also the answer for a PR that only changes the harness and therefore tests nothing by itself: run it once the change is on main.

Why it exists at all: the answer arrived once by accident. Gitea PR #48 corrected a comment in 21 charts, which made all of them "changed" and turned the run into the first all-charts install-test after the move — 19 of 20 matrix charts came up in 12m41s in one job, and exactly one did not (rss-bridge, whose last green run was from a GitHub runner; Gitea #51). Worth having, and nobody had planned it.

• Input charts — space- or comma-separated chart names, empty means all. That is what lets a single chart be tested against the runner, which is what Gitea #51 needs. A name that is not a chart directory, or one listed in ci/excluded-charts.txt, is a loud error: someone typed it, and a green run over zero charts would be the worst possible answer. An empty matrix fails the run for the same reason.
• One throwaway cluster per chart, no reuse. Reuse would save time and change what the test proves — leftovers from the previous install, a warm image cache, fixtures already applied. "Came up from nothing" is the statement.
• Exclusions come from the one parser, ci/run-install-test.sh --list-excluded, and the excluded charts appear in the job summary with their justification, exactly as in the PR pipeline.
• The result is written down. Every matrix leg records its own outcome next to the four facts that decide what the outcome means — date, runner label, K3S_IMAGE, kubectl version — and a final job records the verdict over the whole matrix. A green run nobody notes is unknown again next time. Per leg and not as one table, because a matrix leg only ever sees its own job summary.
• Expected runtime: 20 charts in 12m41s (Gitea PR #48, one job, all legs concurrently on one Docker daemon). The figure is there so a later deviation is visible as one; compare it against the chart count the run prints in its own summary — 24 at the time of writing, and one more with every new chart. The job timeout is 30 minutes per leg and lives in install-test.yml, i.e. it is the same for both callers on purpose.

SonarQube analysis (after the merge, and not a gate)

sonar.yml sends the whole tree through a SonarQube analysis. The workflow itself is the shared block sonar-analysis.yml in workflow-util/pipelines, bound to @v4; the only thing on this side is sonar-project.properties in the repo root, holding the project key jlab-cluster_helm-charts. The key carries the owner because the instance serves two (jlab-cluster and suer), and a project key cannot be changed afterwards without losing the project's history.

What is analysed — the languages were checked against SonarQube's list rather than assumed, and the fit is better than expected:

What is in this repository	Analyser
the 5 shell scripts under ci/ — the whole CI logic	Shell
~30 workflow files	GitHub Actions
238 YAML files and 26 Helm template directories	Kubernetes/Helm, YAML
26 values.schema.json	JSON
the whole tree	Secrets

The last row is the valuable one. The premise of this repository is that no credential ever lands in it — every secret is a 1Password OnePasswordItem reference — and until now nothing checked that premise.

It is not a gate, and that is a decision. The SonarQube Community Build analyses the main branch only; there is no pull-request analysis to hook into, so the trigger is a push to main and a finding appears after the merge. At that point there is nothing left to block, which is why the shared block carries no quality-gate step: a red gate behind the merge would only colour main red without having prevented anything, and a permanent red is a thing people learn to ignore — taking the credibility of validate-charts with it. A gate becomes the right thing under one of two conditions: pull-request analysis is available, or a deployment hangs off the workflow. Until then, read the findings in SonarQube; nothing in this repository turns red because of them.

No exclusions, for the first round. Including the Helm templates, which are not valid YAML because of {{ }}. What the Helm and YAML analysers make of them is only learnable by letting them run over them once, and excluding them up front would discard that answer along with the bulk of the repository. If the first run produces noise genuinely not worth reading, the exclusions go into sonar-project.properties with the finding they answer named next to them — not before.

Prerequisites, both on the instance and neither in a file here: the repository variable SONAR_HOST_URL (a variable and not a secret on purpose — the server address is not a credential, and as a variable it stays readable in the log, which is what makes a wrong one diagnosable) and the secret SONAR_TOKEN, which reaches the block through secrets: inherit. The block guards both and names the setting and its place when one is empty. A missing SONAR_TOKEN never gets that far: the block declares it required, and a missing required secret fails job creation on this instance, so the run would show no steps at all — the same quirk as everywhere else here.

Three things about this path are unmeasured, and the first run after this change is what answers them: whether node24 is in the runner image (the scan action is a JavaScript action declaring runs: using: node24, so a failure will read as an action problem rather than an image one), whether the scanner download gets through — the action fetches the Sonar Scanner CLI from binaries.sonarsource.com at run time, ~59 MiB, and a 17 MB download once failed on this runner for MTU reasons (see "What a GitHub runner had been providing quietly") — and whether a repository variable reaches a workflow called from another repository, which is the same open question the publish route has for OP_CONNECT_HOST. workflow_dispatch is on the hull for exactly this: retrying any of them otherwise costs a dummy commit on main.

Dependency updates (Renovate)

Renovate runs as the shared jlab-renovate-bot instance on the Gitea side, which also serves the other repositories there — see "Renovate on Gitea" below. There is no Renovate workflow in this repo, and none should be added. (The jlab-renovate GitHub App that used to do this is switched off along with the rest of GitHub Actions.) What updates is unaffected either way: the config is renovate.json5 in the repo root (JSON5 for the inline reasoning; requires Renovate >= 41 for managerFilePatterns/customType). There must be exactly one config file — do not add a second renovate.json or .github/renovate.json.

What Renovate keeps current:

What	How
Chart app versions (appVersion: in Chart.yaml)	One customManagers entry per chart, hard-coding the image repository
Sidecar/init images in values.yaml (nginx, guacd, busybox, alpine)	Built-in helm-values manager
All images of freesailarr, including its main one	Built-in helm-values manager — the chart pins every tag in values.yaml and has no app-version block. One exception: mediathekarr is pinned by digest (its tags carry no version) and has its own customManagers block
GitHub Actions	Built-in github-actions manager
Tool versions pinned as workflow env vars (TRIVY_VERSION)	# renovate: datasource=… depName=… comment above the line

Each chart renders its main image as <repo>:{{ .Chart.AppVersion }} — repository in the template, version in Chart.yaml. Renovate cannot join two files, so the mapping lives in renovate.json5, one block per chart. It is deliberately not a # renovate: comment inside each Chart.yaml: that would touch all chart directories and force a version bump plus a publish per chart for a pure comment change. Charts whose tags carry a flavour or date suffix (apache-tika, minecraft-server, rss-bridge, searxng, teamspeak-server) override versioning in their block; the reasoning sits next to each one.

One rule those overrides share, learned from searxng and worth knowing before writing the next one: a capture group is a claim about how the value is compared. Renovate's regex versioning parses major, minor, patch, build and revision with Number.parseInt (modules/versioning/regex/index.js, _parse), so a non-numeric commit hash captured as build becomes whatever digits lead it — 9fea41204 is 9, 777ba8fa4 is 777, and a hash starting with a letter is NaN. Measured on renovate 41.173.1 and 44.48.0, that makes Renovate offer an image pushed earlier the same day as an update to a newer one — a downgrade wearing the clothes of a routine date bump, which is exactly what a review waves through and automerge then carries to main. Parts of a tag that carry no ordering belong outside every group: the anchored pattern still requires them to be there, they simply take no part in the comparison. prerelease is not the way out either — it sorts correctly but marks every tag isStable === false, which hands the whole chart to ignoreUnstable. freesailarr has no app-version block on purpose — see the images exception above; a block would either duplicate the helm-values updates or rewrite an appVersion that selects no image. It does carry one customManagers block, for mediathekarr: that image is pinned by digest because its tags carry no version (latest is a 2024 build, beta moves), and the built-in helm-values manager reads only repository/tag and skips it (skipReason: invalid-value), so the two cannot collide. Digest rules need the same guard as versioning rules and for a sharper reason: while the pin is current, a matching rule and a broken one produce the identical empty result. Verified with two dry-run fixtures — the real digest yields no update, a deliberately stale one yields exactly one digest update back to the pinned digest.

A consequence of pinning every tag in values.yaml: ci/renovate-bump-chart.sh sees an unchanged appVersion and bumps the chart patch. That holds for all ten images alike, the digest-pinned one included — none of them is special — and it is the intended outcome, since appVersion here is the chart's own marker rather than a tracked upstream version.

Two dependencies are constrained beyond their versioning. nginx (the Synapse delegation sidecar) is limited to even minors via allowedVersions. nginx numbers its release branches by minor — even is stable, odd is mainline — and the chart belongs on stable. Patch updates inside the stable branch still flow through normally. The constraint tolerates a variant suffix (1.30.4-alpine) on purpose: docker versioning only ever offers candidates carrying the current tag's suffix, so a suffix-blind constraint would silently freeze nginx the day the sidecar moves to a slimmer base image — a switch to 1.30.4-alpine that #84 has commissioned a check on — with no error and no warning.

linuxserver/qbittorrent (freesailarr) excludes two tags by name, 14.3.9 and 20.04.1. They are leftovers from earlier linuxserver tagging schemes, and docker versioning sorts both above every real release, so Renovate offers them as major updates — PRs #152 and #153 were exactly that. They are named rather than cut off with an upper bound like <10, because a bound would eventually reject a genuine major, while two finished artifacts cannot grow and therefore cannot hide a future release. The same guard as for nginx applies and was run: a lookup from currentValue 4.6.4 still offers 5.2.3 as a major, so the filter does not silently empty the update list. That fixture also shows the cost of leaving it alone — without the rule the very same lookup offers 20.04.1 and never mentions the real 5.2.3, since Renovate fills the major bucket only once.

Not managed: ci/** (install-test fixtures pin throwaway infrastructure to major lines on purpose) and template-chart/** (placeholder image). A consequence worth knowing: ci/matrix-synapse/fixtures.yaml pins a Synapse image that Renovate will not follow — bump it by hand when the chart moves a major.

The version-bump hook

Renovate only rewrites appVersion: or an image tag; it knows nothing about version: <chart-semver>+up<app-version>. Left alone, every chart PR would fail the build-chart version check. ci/renovate-bump-chart.sh <chart-dir> closes that gap and is wired up as a postUpgradeTasks command:

• app major changed → chart major (6.0.0 → 7.0.0), matching "runtime behavior changes" in the semver conventions
• anything else (plain app update, sidecar image) → chart patch
• the +up suffix is recomputed from the new appVersion, dashes normalized to dots

Everything is derived from the base commit, so the script is idempotent across the multiple updates a single Renovate branch can carry.

One-time setup outside this repo: postUpgradeTasks only runs on self-hosted Renovate and only when the command is allow-listed in the instance config:

allowedCommands: ["^ci/renovate-bump-chart\\.sh"]

Without that entry Renovate skips the task without an error and every chart PR stays red on the version check.

If the dependency dashboard reports Failed to look up github-releases package …: no-result for third-party repos, that is the instance's github.com credentials, not this config. The instance needs a read-only PAT as GITHUB_COM_TOKEN for github.com datasource lookups; its own platform token is a Gitea token and is worth nothing there.

Renovate on Gitea

Do not add a Renovate workflow to this repository. It is the obvious thing to reach for — Gitea has no app model, so a scheduled job in the repo looks like the only way to run the bot — and it is wrong here: a shared jlab-renovate-bot instance runs outside this repository and has it enrolled. A second runner against the same repo produces duplicate PRs and two schedulers rewriting each other's branches, which is not a safety net but a fault. This paragraph exists so the next reader who finds no workflow knows why instead of building one.

The instance is outside this repository, which has two consequences worth writing down:

• It reads our renovate.json5 and the per-chart rules do fire. Measured, not assumed: its first run here opened PRs for searxng and gotenberg plus the dependency dashboard. Those two are customManagers targets of ours — no built-in manager finds an appVersion: in a Chart.yaml — so the config is demonstrably in effect.
• allowedCommands is already granted there, and the version-bump hook runs. This was the expensive unknown: postUpgradeTasks only executes when the command is allow-listed in the instance config, and a missing entry is skipped silently, leaving every chart PR red on the version check. The proof is in that searxng PR's diff, which carries version: 1.0.4+up2026.9.11.ffe96f8a6 → 1.0.5+up2026.9.13.96479fff6 alongside the appVersion change. Nothing but ci/renovate-bump-chart.sh writes version:. The allow-list is not visible from this repo, though, so if it is ever dropped the symptom here will be red chart PRs with no explanation.

One item is easy to miss and is named in "The version-bump hook" above: the instance needs a GITHUB_COM_TOKEN for github.com datasource lookups. Every github-releases/github-tags lookup and the whole built-in github-actions manager resolve against github.com, the platform token is a Gitea one and worth nothing there, and unauthenticated lookups hit the rate limit inside a single pass. The symptom is a no-result line on the dependency dashboard, not an error.

PR policy

The config is aligned with the one in cluster-config, because the same jlab-renovate-bot instance serves both and two repos that behave differently under one bot is a source of surprises rather than of safety.

• Automerge is global (automerge: true, automergeType: "pr", platformAutomerge: true) — every update type, majors included. The safeguard is not the absence of automerge, it is the branch protection: main requires a review and requires validate-charts to pass. platformAutomerge arms the PR on the forge, and Gitea merges it when both conditions are met, so an armed PR without a review simply sits there. This is the setting the older platformAutomerge: false comment in renovate.json5 named as the better one "once required checks exist on main" — they now do. Same reasoning as cluster-config, which retired its dashboard-approval step for it.
• minimumReleaseAge: "3 days" for patch, minor and digest updates. Renovate does not even open the PR before then. A merge here is never just a merge — a chart publish follows it straight away — so a release that gets yanked hours after it appeared must not reach the registry. Majors carry no age gate: they are held at the review, not by a delay.
• Majors arrive one line at a time (separateMultipleMajor): a v1 → v3 jump produces a PR for the latest v2 first and then one for v3 — the stepwise procedure the versioning section requires. separateMinorPatch: true likewise splits minor and patch into separate PRs, so one red install-test does not block unrelated updates.
• No PR caps (prConcurrentLimit: 100, prHourlyLimit: 0), as in cluster-config. The known cost, written down rather than discovered: the first run after a quiet week can open many PRs at once, and every chart PR starts a k3d install-test — which on a Gitea runner queues inside one job rather than fanning out (see "Where this repo lives"). That queue is the accepted trade for not having updates trickle in four per hour.
• config:best-practices is in extends. Two of its members matter here: helpers:pinGitHubActionDigests moves the workflow files from actions/checkout@v7 to SHA pins (wanted), and docker:pinDigests is switched off again for custom.regex and helm-values — an appVersion: is a tag by construction (<repo>:{{ .Chart.AppVersion }}, and the +up suffix is built from the same string), so a digest has nowhere to go. Sidecar digests in values.yaml could be pinned; that is a separate decision, since each digest change would be a chart patch plus a publish.
• The manually dispatched change-app-version.yml still exists for one-off bumps, but the normal path for app updates is now a Renovate PR.

Creating a new chart

The wiki carries the same procedure as a worked checklist: Creating a new chart. A new chart also needs a wiki page of its own before its PR is merged.

1. Copy template-chart/ to <chart-name>/ and replace every xxx placeholder: template file names (xxx.deployment.yaml → <app>.deployment.yaml), Chart.yaml, the values.yaml keys, the .Values.xxx.* references inside the templates, and both the xxx. helper prefixes and the values paths in values.schema.json. Then walk the six points in "Values that must not lie" below — the schema and the null-safe fallbacks are copied with the template, but their content is chart-specific and wrong until you adapt it. The same goes for templates/NOTES.txt: it comes with the template as a worked example and is placeholder text until you replace it with what an operator of your app needs after the install (see "Everything else" below); its leading {{/* … */}} block explains the bar and is meant to be deleted.
2. Add .github/workflows/publish-<chart-name>.yml (copy an existing one, e.g. publish-outline.yml).
3. Add the chart name to the choice list in .github/workflows/change-app-version.yml.
4. Add a customManagers block for the chart's main image in renovate.json5 (copy a neighbouring one, swap managerFilePatterns and depNameTemplate) — without it the chart's app version is never updated. Check the image's tag list first: if it carries a flavour, date or prerelease suffix, add a versioningTemplate with a comment saying why. Floating tags belong outside the pattern: ytdlp-web-player anchors its versioning regex on the v prefix precisely so latest and stable — both of which move — can never be selected.
5. Only if the chart needs a dependency: read "Chart dependencies" in the style guidelines first, it is a bar rather than a formality. Nothing has to be added to the pipelines — ci/helm-dependencies.sh already runs everywhere and keys off the dependencies: block — but the subchart's own updates then depend entirely on Renovate's built-in helmv3 manager plus a chart-version bump, which is the part to write down at the pinned version.

Validate locally with:

helm lint --strict <chart-name>
helm template <chart-name> --set ingress.domain=example.com --set ingress.clusterIssuer=letsencrypt

(required values vary per chart)

For runtime verification (do the workloads actually start, do probes/security settings hold?), use a throwaway local k3d cluster: k3d cluster create <name>, apply a stub CRD for OnePasswordItem plus dummy secrets (and throwaway postgres/redis pods where the app needs them), helm install the chart, watch pod readiness/events, then k3d cluster delete <name>.

Chart style guidelines

These guidelines are binding for every current and future chart in this repo. They are the consolidated end state of the conventions overhaul (#32, batches 0–4); template-chart/ is the reference implementation. The wiki mirrors them as the chart style guide, with the incident behind each rule collected in Lessons learned — change a rule here and update that page in the same PR.

1. Naming

• Resources are named {{ .Release.Name }}-{{ .Chart.Name }}-<kind> (e.g. ...-deployment, ...-sfs, ...-service, ...-secret, ...-configmap, ...-ingress, ...-pvc).
• Template files are named <app>.<kind>.yaml; stateful apps use a StatefulSet in a <app>.sfs.yaml template instead of a deployment.
• Documented exception — existing PVC names: PVCs of already-published charts keep their historical names even where they deviate from the scheme. Renaming a PVC makes Kubernetes provision a new, empty volume, so a rename would orphan the existing data. New charts name PVCs {{ .Release.Name }}-{{ .Chart.Name }}-pvc per the scheme.
• StatefulSet renames are possible (data survives because the charts use standalone PVC templates, not volumeClaimTemplates) but only ship in a major-bump PR with a one-time migration note: kubectl delete statefulset <old> --cascade=orphan (and delete the leftover old pod) before upgrading.

2. Versioning

• Chart.yaml uses version: <chart-semver>+up<appVersion>; the +up suffix matches appVersion, with dashes normalized to dots (e.g. 0.0.36-beta → +up0.0.36.beta). appVersion is the exact image tag of the main app image.
• Semver rules for the chart part:
  • Patch: bug fixes without behavior or values-schema change; plain app-version updates.
  • Minor: additive, backward-compatible changes (new optional values with unchanged defaults).
  • Major: runtime behavior changes or breaking values changes (e.g. resource renames, probes, security hardening, values that become required).
• Any push to main touching a chart directory publishes that chart, so the version must be bumped in the same change.

3. Images

• The main app image tag comes from {{ .Chart.AppVersion }} — never hardcoded.
• Documented exception — freesailarr: the chart bundles ten independently versioned upstreams (nine of them enabled by default), so none of them is "the" app version. No tag comes from appVersion; every image is pinned as <app>.image.repository / <app>.image.tag in values.yaml, which is also what puts them under the built-in helm-values manager. Its appVersion is the chart's own version marker, freely chosen and tied to no upstream — raised by hand when a new edition of the chart is marked as a whole. Consequently the chart has no app-version block in renovate.json5 and is not in the change-app-version.yml choice list. mediathekarr adds a second deviation on top: its tags carry no version at all, so it is pinned by image.digest as well and is the one image in this chart with a customManagers block — a repository/tag pair alone would leave it without any update path.
• Documented exception — minecraft-server: the only chart whose main image tag can be overridden, through image.tagOverride (#158, empty by default — the tag still comes from appVersion). appVersion is the tag and therefore fixes the JVM, and since chart 5.0.0 only the newest Java line is maintained out of this one directory; the older lines stay published as frozen chart versions (java8 and java17 at 4.0.0, java21 at 4.1.0) and never receive another change. Without the override, running an older JVM would mean installing a frozen chart and giving up every later fix — so the line is a decision at rollout time rather than a property of the chart. The override is named tagOverride, not tag, because it decouples the running image from the chart version, and it is deliberately outside Renovate's reach (renovate.json5 tracks the appVersion in Chart.yaml and nothing else). The chart checks the pairing itself: the Java line of the effective tag against minecraft.version, failing the render with both values named — a mismatch used to install silently and only break the container afterwards.
• Auxiliary images (sidecars, init containers, secondary workloads — e.g. the delegation nginx in matrix-synapse, guacd in patchmon, busybox init/consume images in paperless-ngx) are pinned via values instead of being hardcoded in templates, and their pinned versions are reviewed alongside every app-version update round.

4. Ingress

• ingress.domain and ingress.clusterIssuer (cert-manager) are required values without defaults — every installation sets both explicitly; they are the only values needed for a default install.
• ingress.className selects the ingress class and defaults to nginx.
• ingress.annotations takes optional extra annotations that are merged with the cert-manager cluster-issuer annotation the chart always sets.
• The TLS secret is named tls-<release>-<chart>-ingress.
• Documented exception — homeassistant: keeps its upstream-style flexible ingress values (host/tls lists, no required values); only its ingress.className default follows the standard (nginx, clearable to omit the field).
• A shared domain with sub-paths is the second admissible ingress shape. Where a chart exposes several apps, they may either each take a host of their own or share one host and be told apart by path (domain.tld/sonarr, domain.tld/radarr, …) — one DNS name and one certificate for the whole bundle. The rules for the shared shape: the domain is a single value (ingress.host), each app's path is its own value defaulting to /<app>, the paths render in one Ingress object <release>-<chart>-ingress with the standard TLS secret tls-<release>-<chart>-ingress, and an app that carries a host of its own always wins and keeps its own object — so both shapes mix without extra values. An app that needs a per-path proxy annotation (rewrite-target is per object, not per path) gets its own object on the same host, repeating the same TLS secret name and omitting the cert-manager annotation, so that exactly one certificate is requested. Apps that cannot serve a base path must be rendered at / or refused with a fail — never rendered somewhere they provably cannot work.
• Documented exception — freesailarr: the chart bundles several independently exposed apps in one pod, so it has no ingress.domain. It offers both shapes. By default each app carries its own <app>.ingress.enabled plus a required <app>.ingress.host (required only while that app's ingress is enabled and it is not on the shared domain) and renders its own Ingress object <release>-<chart>-<app>-ingress with its own TLS secret tls-<release>-<chart>-<app>-ingress. Setting the global ingress.host switches every app without a host of its own to the shared-domain shape under <app>.ingress.path; qBittorrent (no base path, needs a prefix-stripping rewrite) gets the extra object described above, and seerr as well as mediathekarr are pinned to / — mediathekarr because its UI links assets with root-absolute paths, which a rewrite cannot fix; with both of them on one shared host the duplicate-path guard fires and one of them needs a host of its own (its ingress is therefore off by default). clusterIssuer, className and annotations stay global defaults under ingress. and are overridable per app.

5. Security hardening

• Defaults on every workload: automountServiceAccountToken: false, non-root user, readOnlyRootFilesystem: true, all capabilities dropped, allowPrivilegeEscalation: false.
• Pod and container contexts are values-configurable via securityContext.pod / securityContext.container.
• Writable paths (e.g. /tmp, nginx cache dirs, s6-overlay's /run) are emptyDirs so the root filesystem stays read-only.
• Where an image cannot satisfy the full standard (e.g. Home Assistant must start as root for s6-overlay, paperless needs a root init container to hand over /run, samba needs its capabilities), the deviation is minimal and must be justified with a comment in that chart's values.yaml.

6. Probes

• Every workload has liveness, readiness and startup probes (documented exception: the paperless-ngx consume CronJob).
• Probes are values-configurable with calibrated defaults (the startup probe gates liveness/readiness; its budget is documented in a values comment per chart).

7. Resources (auto-limits)

• Resource requests ship as chart defaults in values.yaml and are user-overridable.
• Limits are computed by a per-chart _helpers.tpl helper (reference implementation in template-chart/templates/_helpers.tpl): an explicitly set limit always wins, missing memory and ephemeral-storage limits default to resources.limitFactor × the request (factor default 3), and a CPU limit is only rendered when set explicitly — otherwise CPU stays unlimited (it is compressible, throttling instead of OOM-killing). The helper is copied into every chart rather than shared through a library chart — which would be a second artifact to publish, version and pull for a helper that changes about once a year; the resource-quantity render test above exists because a copy can carry an old bug into a diff nobody reads.

8. Secrets

• Secrets are 1Password OnePasswordItem CRs: values expose secrets.<name>SecretRef holding an op:// item path; the 1Password operator materializes the Kubernetes Secret in-cluster.

9. Config changes must restart the workload (#82)

• Every workload that consumes a ConfigMap carries a checksum/config annotation on its pod template — one line per ConfigMap it consumes:

  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/<chart>.configmap.yaml") . | sha256sum }}
  

  Reference implementation: template-chart/templates/xxx.deployment.yaml and xxx.sfs.yaml. Where a workload consumes several ConfigMaps, use one annotation per file with a distinguishing suffix (checksum/config-<name>).

• Why it is mandatory, not optional: without it a changed ConfigMap does not change the pod template, so nothing restarts and the running container keeps the old config — silently. subPath mounts (the usual form here) are never refreshed by the kubelet at all, and most apps read their config only at start. In #82 this produced a real outage: the router kept routing to Services that no longer existed while every bundle was green and the pod was Ready.

• The annotation is content-based, not time-based: the hash is taken over the rendered ConfigMap, so an unchanged config renders an identical hash and helm upgrade is a no-op. It never degenerates into a restart-on-every-upgrade.

• Adding the annotation to an existing chart is a major bump (§2): runtime behavior changes — a config change now costs a restart. Name that cost in the PR (some workloads take minutes to come back).

• Secrets from OnePasswordItem cannot use this mechanism. Helm never sees their content — the 1Password operator fills the Secret in-cluster after rendering, so there is nothing to hash. A chart that mounts such a Secret as a file documents the limitation in its values.yaml where it cannot close the gap.

• What can be covered is the secret's op:// path, via a separately named checksum/secret-ref annotation over the rendered OnePasswordItem:

  checksum/secret-ref: {{ include (print $.Template.BasePath "/<chart>.secret.yaml") . | sha256sum }}
  

  Repointing secrets.<name>SecretRef at a different vault item then takes effect instead of silently leaving the old credentials in the running pod. The separate name is deliberate: checksum/config promises "the config changed", checksum/secret-ref only "the config source changed" — merging them into one annotation would overstate the weaker guarantee. Reference implementations: minecraft-server, homeassistant.

• operator.1password.io/auto-restart only works for Deployments — do not set it on a StatefulSet. Verified against onepassword-operator 1.12.0 (pkg/onepassword/secret_update_handler.go): restartWorkloadsWithUpdatedSecrets enumerates exactly one workload type, appsv1.DeploymentList, and getPodTemplate accepts only *appsv1.Deployment. The RBAC role grants statefulsets, and v1.11.0 generalized the restart code (restartWorkload, pod-template stamping via operator.1password.io/last-restarted), but the PR that would add StatefulSets is still open — so the annotation on a StatefulSet is a silent no-op. Setting it there is worse than not setting it: it advertises a guarantee that does not exist. Affected charts (matrix-synapse, samba, homeassistant) instead document the manual kubectl rollout restart statefulset <release>-<chart>-sfs. Re-check this when the operator version is bumped; a Deployment-based chart may set the annotation.

• If a Deployment-based chart ever does set it, mind the precedence: the annotation on the OnePasswordItem is copied onto the Kubernetes Secret and read first, so a false there overrides a true on the workload — the opposite of what the operator's usage guide suggests. Lookup order is Secret (from the OnePasswordItem) → workload → namespace → the operator-global AUTO_RESTART env var. The trigger is the 1Password item's version counter, not its content, so a re-save with identical values still restarts.

10. Environment variables

• Every workload container offers an additive env value block, appended to the env entries the chart always sets.
• Entries are rendered through tpl (so values may contain Helm templating) and support the value, secretKeyRef and configMapKeyRef forms (pattern: template-chart/templates/xxx.deployment.yaml).
• Additive means additive: an entry repeating a name the chart already sets does not replace it, and what the pair does depends on when it arrives. At install both entries reach the pod and the later one — the user's — wins. On an existing release the same entry is lost: Kubernetes merges env by name, the pair collapses to the chart's own value, and helm upgrade still reports success. Measured on urlaubsverwaltung (helm 3.19.0, Kubernetes 1.35): helm get manifest shows the variable twice while the live Deployment shows it once, carrying the chart's value. The failure mode is silence, so a chart whose NOTES.txt tells operators to override one of its own variables has to say that the override only takes at install time — and that a green upgrade proves nothing, the value belongs read back off the Deployment.

11. Scaling

• Every Deployment/StatefulSet has a replicas value (default 1, the values path per workload matches its section) that also accepts an explicit 0 for per-workload fine control.
• Every chart has a global scaleDown flag (default false): scaleDown: true takes precedence over all replicas values, renders every Deployment/StatefulSet with replicas: 0 and suspends CronJobs. It stops the app but does not uninstall it — PVCs, Secrets, Services and Ingress stay in place; setting it back to false brings the workloads back up.
• Implemented via a per-chart <chart>.replicas helper in _helpers.tpl (reference in template-chart); the helper uses an explicit nil check (kindIs "invalid") because Helm's default would swallow an explicit 0.

12. Values that must not lie (#68, #93, #99, #215)

Every chart ships a values.schema.json with additionalProperties: false on every chart-owned object level — not just the top one. The gap this closes is a key that looks like configuration and does nothing: the pilot (#68) found three of them in one chart, one of them nested three levels deep and quietly leaving a game world unpinned. Objects that are passed verbatim into the pod spec stay open (probes, securityContext.pod/.container, global, free-form annotation maps): Kubernetes owns their schema, and restricting them would only reject fields that do not exist yet. The schema adds to required(), it does not replace it.

Reference implementation: template-chart/values.schema.json and template-chart/templates/_helpers.tpl.

1. A value read outside its obvious template needs its own guard. Before putting an Ingress behind ingress.enabled, check whether anything else reads ingress.domain — in templates/ and in values.yaml, because env entries are rendered through tpl and hide such reads. If something does, that read needs its own required(); the gate then frees clusterIssuer but not domain. This happened in six charts, and matrix-synapse shows why it matters more than the count: its delegation nginx serves ingress.domain in the /.well-known/matrix/{server,client} endpoints through which federation finds the homeserver. Gating without the guard publishes

{"m.server": ":443"}     {"m.homeserver": {"base_url": "https://"}}

— rendered cleanly, pod green, federation broken. Check per chart and in both directions: in calibre-web-automated nothing outside the Ingress reads the domain, and there the gate frees both values.

2. Probes, security contexts and resources go through the null-safe helper, never behind an if. An override written without children is YAML null and erases the block: the container renders securityContext: null (unhardened, healthy, silent), a probe disappears, or resources loses its requests and computed limits. The defaultedDict helper defines the opposite semantics — an empty block means "chart defaults apply" — while an explicitly set value still wins, including false and 0 (hence the kindIs "invalid" check; default and mergeOverwrite both swallow zero values). Do not wrap these blocks in {{- if }}: a guard does not render a visible null, it removes the probe entirely, which is the same bug with no evidence left.

The rule is about the shape, not about those three block kinds. satisfactory shipped 7.0.0 with game null-safe and its two neighbours not (Gitea #16): --set server=null aborted the render, and a single erased key was the quieter half — a nulled server.gamePort rendered port: and nodePort: empty into the Service and an empty containerPort into the StatefulSet, rejected by nothing until the API server saw it. Every chart-owned block a template dereferences belongs behind the helper, and a chart where only some of them do is worse than one where none do: whoever sees game=null fall back cleanly assumes server=null does too.

A deliberate exception is fine; an undocumented one is the bug. satisfactory.storage is left un-defaulted on purpose — an erased size yields an invalid PVC that fails at apply time rather than a volume quietly sized by the chart — so server: null still renders a PVC without a size. That is a defensible trade and it is not the null-safety bug next to it; what makes it safe to leave standing is that the schema, values.yaml and the chart page all name it as an exception and point at the issue where it is argued (Gitea #39). Note also where the trade actually bites: only on a new install. For an existing release a fallback to the default size would fail too, because a PVC cannot shrink. Do not overturn a reasoned exception inside a bugfix patch — the patch then stops being one.

And helm template does not render NOTES.txt. A value read there — server is read in four files in that chart — stays broken after a fix that looks green in every render, until someone runs a real helm install. helm install --dry-run=client is not the way around it either: it still wants a reachable API server (measured on helm v3.22.0). What works without a cluster: in a throwaway copy of the chart, wrap the verbatim NOTES.txt body in a {{- define }} and include it into a scratch ConfigMap, which helm template does render — same engine, same root context, only the sink differs.

3. Do not hang liveness on an external dependency where a dependency-free path exists. zipline deliberately probes / for liveness and /api/healthcheck only for readiness and startup, so a database outage cannot restart the pod. Where no such path exists — outline returns 500 on / when the database is down — all three probes may share the endpoint, but the reason belongs in a comment on the helper, or the next reader will "fix" the inconsistency.

The other half of the same rule: readiness must hang on the dependency, and a separate path is not proof that it does. urlaubsverwaltung probed /actuator/health/liveness and /actuator/health/readiness for two years and still stayed 1/1 Ready through a total database outage (#218), because Spring Boot's readiness group contains readinessState alone — the split paths were right, the group content was empty of anything external. Whatever aggregates behind a probe path (Spring health groups, a /healthz a framework assembles for you) has to be read off the running endpoint, not off the path name; the chart names the members explicitly (MANAGEMENT_ENDPOINT_HEALTH_GROUP_READINESS_INCLUDE=readinessState,db) so the composition is in the template rather than in an upstream default.

4. A fallback reproduces this chart's defaults, not the repo standard. Five charts deviate from the hardening baseline on purpose. A "cleaned up" standard default is not tidying, it is an outage: calibre-web-automated must keep readOnlyRootFilesystem: false, because in read-only mode the image disables its PUID/PGID remapping and the runtime uid silently moves from 1000 to 911 — making existing volume data inaccessible. The same applies to non-standard values that look like mistakes: satisfactory keeps limitFactor: 2 (that is what holds the memory limit at the published 8Gi) and fsGroupChangePolicy: OnRootMismatch (which avoids a recursive chown of multi-GB game files on every start).

5. A probe default carries the probe's context, not just its timings. Two different symptoms, both from real charts: teamspeak-server addresses its TCP port by the name file-transfer because the voice service is UDP and cannot be probed — a default without that name leaves the workload with no working probe at all. patchmon sends a Host header because the server answers 403 to any request whose Host is neither loopback nor a CORS_ORIGIN host — a default without it means the pod never becomes Ready. Port names, Host headers and exec commands belong in the fallback; where the context is dynamic, pass the root context in and rebuild it (patchmon derives the header from ingress.domain).

6. A value that reads as the negation of the variable it feeds must actually be negated — and not cannot do it. satisfactory shipped game.seasonalEvents straight into the image's DISABLESEASONALEVENTS (#215): the default true switched the events off in every install for the chart's whole life, while the schema described the same key as "true means events are enabled". The fix is the loud one — rename the key so it says what it does (game.enableSeasonalEvents) rather than silently inverting the template under existing releases, because the old key then fails schema validation instead of quietly meaning the opposite. Two mechanics come with it: Go's not treats every non-empty string as true, so not "false" is false and the string form these schemas deliberately accept flips the wrong way — compare explicitly (eq (lower (printf "%v" $v)) "true") instead. And a value that cannot be read as either boolean belongs in a fail, not in an implicit branch: enableSeasonalEvents: "yes" must abort the render, not be quietly filed under "off". When such a rename lands, the new default is the one that reproduces what the chart already rendered; changing the meaning and the behaviour in the same release turns the loud fix back into a silent one.

Two things about the mechanics that are easy to lose:

• null reaches the schema in two cases out of three, and which one you are in depends on the chart's default, not on what the user wrote. Measured on Helm 3.22.0 with a probe chart, because the wording here was wrong before and was applied to exactly the case it got wrong:

  • a user null over a non-null chart default (--set x=null, or x: null in a values file) is treated as a deletion and the key is removed before schema validation — the schema never sees it, and only the helper can catch it;
  • a null chart default survives into validation with or without a user null on top of it, because there is nothing to delete it against, and the schema sees the key present with value null;
  • a key without any default (e.g. resources.limits) likewise survives, and a strict "type": "object" aborts the render.

  That is why the helper and the widened ["object", "null"] / ["string", "null"] types are both needed: each covers a case the other cannot. Everything else stays strictly typed.

• A value that has no valid default cannot be enforced by values.schema.json at all, and required() is the whole enforcement. The chain is short and it has no exit (#18): the key ships as null, so it is present at validation time, and JSON Schema's required tests presence and not null-ness — the array never fires, under any input. The two ways to make it fire both break the rule that helm lint --strict passes on chart defaults, since lint validates those defaults against the schema: dropping the null default gives missing property '<key>', and narrowing ["string", "null"] to "string" gives got null, want string — each a red validate-charts run on a chart nobody touched. So such a key is guarded with required() in the template and described as required in the schema; a required array next to it would be a guard that cannot fire, which is the same defect as a value that does nothing, dressed as a safeguard. This is why no chart in this repo carries one.

• Validate every schema against the real values, not only against chart defaults. Defaults are valid by construction, so they prove nothing. Render each chart against its bundle in cluster-config/fleet-cd/ and against helm get values — the two differ more often than expected (four of ten bundles in one wave, always with the repo file being the worse one). This step caught a schema of mine that would have broken three live paperless releases, because all three write tikaGotenberg.enabled as the string "true" and the first draft demanded a boolean. Result per chart is a list of rejected keys, which is the cluster side's work order before the version is pulled up.

13. Everything else

• icon: is mandatory in every Chart.yaml. Documented exception — freesailarr, the only chart without one: the rule everywhere else is "the icon is this chart's one upstream's logo", and a chart bundling ten independently versioned projects has no such upstream — picking one of the ten would imply a "main app" the stack deliberately does not have. That is the whole criterion, and it applies to no other chart in the repo: a single-upstream chart without an icon is a missing icon, not an exception. The omission costs an INFO line in helm lint --strict (icon is recommended), which still exits 0.
• helm lint --strict must pass without findings.
• Every chart ships a templates/NOTES.txt (#43), template-chart included — so this is a requirement for a new chart, not a nice-to-have it can defer. No list of names here on purpose: a count goes stale the moment the next chart lands, which is exactly how the previous wording came to claim a number it never matched. Reference implementation: template-chart/templates/NOTES.txt, a placeholder-carrying template rather than finished prose, so a new chart adapts it instead of starting from an empty file. The bar it sets: print what an operator would otherwise have to look up in the templates (the rendered access URL, which fields the 1Password item must carry, the options actually active in this release, and the chart's known traps), conditional on the values really set. A NOTES.txt that only repeats the values is skimmed once and never read again. No real domain anywhere — the URL renders from ingress.domain, and where an example is unavoidable it is example.com. A NOTES.txt renders no Kubernetes object, so writing or reworking one is purely additive: a minor bump.
• CI: every installable chart has fixtures under ci/<chart>/ (test-values.yaml, optionally fixtures.yaml/fixtures.env.yaml/settings.env) or a justified entry in ci/excluded-charts.txt — the install-test fails if both are missing.

14. Chart dependencies

Until ytdlp-web-player (#214) this section read "charts have no dependencies (charts/ directories are empty)". That rule is gone, replaced by a bar rather than a ban — a dependency is admissible, not free, and the reason it stopped being a ban is worth keeping: the chart needed an oauth2-proxy in the data path in front of an application that ships no authentication at all, and proxy and app are only useful wired to each other.

When a dependency earns its place. All three, not two of three:

1. The dependent piece would otherwise be a second Helm release that only exists to serve this one, with the wiring between them living in neither chart.
2. Upstream is the right owner of it. We consume their chart as it comes: we do not patch it, do not re-harden it, do not re-implement its values. Where their defaults fall short of the guidelines above, that is documented in our values.yaml, not fixed by us — and a values-level fix stays available to whoever installs it.
3. It is optional and off by default (condition: in Chart.yaml), so the chart is complete without it and a default install renders none of its objects.

What it costs, all of it visible before the next one is added:

• The subchart is invisible to everything downstream. Its version is inside our packaged .tgz, so anything that watches Helm release versions — Fleet's fleet.yaml in cluster-config included — sees only our chart version. A new upstream release, security fix included, reaches a cluster only when we publish a new version of the outer chart. Nothing reports the gap. Renovate's built-in helmv3 manager opens the PR (no customManagers entry needed — the repository URL and version sit together in one file) and ci/renovate-bump-chart.sh bumps the chart patch that ships it; that chain is the entire safety net. State this in a comment at the pinned version, not only in a review.
• Every pipeline that touches the chart needs a network fetch first. helm lint, helm package, helm template and helm install all exit 1 while a declared dependency is missing from charts/, so ci/helm-dependencies.sh runs before each of them (see CI/CD above). One of those call sites is the resource-quantity render test, which runs over every chart on every PR — a dependency-carrying chart can therefore turn a third-party Helm repository's bad day into a red pipeline on a PR that touched nothing related.
• charts/*.tgz and Chart.lock are not committed (.gitignore says why): the archive is a third-party binary nobody reviews in a diff, and a committed lock would arrive stale because Renovate rewrites Chart.yaml and cannot regenerate it — which is also why the pipelines run helm dependency update and not helm dependency build.
• The repo's own mechanisms stop at the subchart boundary. checksum/config, the 1Password auto-restart annotation and the scaleDown flag reach our templates only. A chart that renders a OnePasswordItem for its subchart documents the manual kubectl rollout restart that replaces the missing restart-on-change.

Two implementation notes that cost time otherwise: a values key containing a dash is unreachable as .Values.oauth2-proxy (that parses as a subtraction) and needs index .Values "oauth2-proxy", while condition: in Chart.yaml splits on dots only and needs no escaping. And inside a subchart .Chart.Name is the subchart's name — a value templated into it (config.configFile in this case) must spell the parent chart out literally; only .Release.Name is shared.

Charts that need no dependency still have none, and charts/ stays empty for all of them. template-chart remains dependency-free: the scaffold should not carry a dependencies: block a new chart has to remember to delete.