jlab-cluster/helm-charts
3
0 Fork 0
Code Issues 15 Pull Requests 3 Actions Packages Wiki Activity
Page: CI CD
Pages
CI CD Chart Style Guide Charts Forge and Tooling Home Lessons Learned Local Development New Chart Renovate Versioning Workflow apache tika bentopdf bulwark calibre web automated convertx cyberchef freesailarr garage gotenberg homeassistant keycloak matrix synapse minecraft router minecraft server outline paperless ngx patchmon rallly rss bridge samba satisfactory searxng stalwart sure teamspeak server template chart urlaubsverwaltung valkey voucher vault ytdlp web player zipline
Clone
31
CI CD
Justin Klein edited this page 2026-10-15 17:51:52 +00:00
Unescape Escape
Table of Contents
This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

CI/CD

Two pipelines: one that validates a pull request, one that publishes a chart after the merge. Both run on Gitea Actions and read the files under .github/workflows/ unchanged. Next to them two runs that no pull request starts: the all-charts install-test, dispatched by hand, which asks whether the environment under an unchanged chart still holds — and the SonarQube analysis, which runs on every push to main and blocks nothing.

Which of the four can stop a merge is the distinction to carry into every section below: only the PR pipeline can. The publish runs after the decision, the all-charts install-test runs when someone asks, and the analysis reports a state that is already merged.

Publishing

  • publish-<chart>.yml — one thin workflow per chart (25 of them). It triggers on pushes to main touching <chart>/**, or manually, and calls the shared block below.
  • A push to a chart directory on main publishes that chart. The chart version must be bumped in the same change; the PR pipeline enforces it.
  • publish-helm-chart.yml — not a workflow of this repository. It lives in workflow-util/pipelines, the shared building-block repo, and does the same as before: fetch dependencies, helm package, then a single HTTP POST of the .tgz to the registry.

The registry is a classic, index-based Helm registry (Gitea's /api/charts endpoint), not an OCI one: helm push speaks only OCI and is not used, there is no helm registry login session, and every request carries its own Basic auth. Credentials and the target URL come from 1Password Connect (item jlab-registry-access with username, password, charts-url).

What a caller here supplies

The hulls are bound to a tag, never to that repository's default branch — otherwise a change over there would change publishing here, unannounced, and the failure would appear where nobody touched anything. Each hull is four lines of substance:

uses: workflow-util/pipelines/.gitea/workflows/publish-helm-chart.yml@v3
with:
  chart-name: '<chart>'
  registry-owner: ${{ github.repository_owner }}
secrets: inherit
What Why it is the caller's job
registry-owner A required input with no default. The owner used to be derived inside the workflow; as an input the expression is evaluated here, where the value is measured. That repository's house rule is that a shared block never guesses an owner segment — the rule was bought by a hardcoded owner that sent every project's images to one wrong account.
secrets: inherit The block needs OP_CONNECT_TOKEN from this repository. Unchanged by the move.
Variable OP_CONNECT_HOST A repository variable, set on the instance. It replaced the Connect host that used to be hardcoded in our copy; a shared block must not carry one tenant's infrastructure address. The block's first step fails and names the variable when it is empty.
ci/helm-dependencies.sh Still our script, and still run before helm package. The block's checkout clones the calling repository, so it runs from this tree — which also means deleting or renaming it breaks publishing from the other side of the tag.

Where the reasons live now. Why the credentials go in through curl --config - rather than --user, why --fail-with-body is what makes a failed upload fail the job, why the file name is built rather than globbed, and how the loaded secrets are read: all of it is on the publish-helm-chart page of that repository's wiki. It is deliberately not repeated here — two copies of one explanation means one of them is wrong eventually, and the file those sentences describe is no longer ours. Nothing was lost in the move: the upload step's shell body went across unchanged apart from two error messages that name the owner as an input.

The target URL is composed, not stored

charts-url holds only the base up to /api/packages. The workflow appends the rest:

op://GitHub-Actions/jlab-registry-access/charts-url  =  https://<host>/api/packages
target                                              =  <base>/<owner>/helm/api/charts

The owner segment is the registry-owner input, which every hull fills with github.repository_owner — verified to be populated on this Gitea instance (measured as jlab-cluster on a temporary probe step, since removed). It used to be read inside the workflow; passing it in from here changed nothing about the resulting URL, only about where the expression is evaluated. Consequences worth knowing:

  • No registry address appears anywhere in the repository, not even the owner path segment: it is derived from the repository rather than stored. The thin callers need no setting of their own beyond the one line.
  • The second owner on the same instance (suer, for suer-configurator) gets the right target from the same block, without a second value to keep in step. Since the owner is an input rather than a derived value, that repository can use the block as it is.
  • On the github.com mirror the owner is a different one — but Actions are switched off there, so the publish never runs. If it ever did, the owner would be one the instance does not know: HTTP 404, loud, not a silent upload to the wrong place.

What has to be in the vault item, because the block rejects anything else — the base only, no owner, no package type, no endpoint:

op://GitHub-Actions/jlab-registry-access/charts-url  =  https://<host>/api/packages

The block guards both halves of the composition and names the vault field in the error, because the only other signal would be a 404 against a URL 1Password has masked. The individual guards and the measurements behind them are on its wiki page; what matters on this side is the shape above.

The migration to that shape was a hard cut (historical, done): the vault item was shortened to the base first, then the workflow change was merged. Both possible mismatches fail with a 404 and neither publishes to a wrong place.

Proving the path after a change to the publishing route

A change to the hulls or to the shared block touches no chart directory, so merging it triggers no publish and leaves the new route unproven until the next chart change happens to exercise it.

The end-to-end check is a single manual workflow_dispatch of any publish-<chart>.yml — this is what those dispatch triggers are for, and after the move they are the only way to start a publish by hand. The registry replaces an already published version on re-upload (HTTP 201, below), so it is a safe test of the whole path. Only the maintainer can trigger it; the bot account cannot dispatch workflows.

Three things were listed here as only that run could prove, and they were the whole remaining risk of the move: that a cross-repository uses: resolves on this instance, that a repository variable reaches a workflow called from another repository, and that the upload itself still lands.

The first two are now proven, and not by this route. The SonarQube analysis is the same shape — a hull in this repository calling a block in workflow-util/pipelines and reading a repository variable inside it — and its first run (run 375, a push to main) got as far as a running scanner. That could not have happened unless the cross-repository uses: had resolved and vars.SONAR_HOST_URL had arrived: the block guards the variable and exits before the scan when it is empty. Same forge version, same mechanism, so the answer carries over to OP_CONNECT_HOST in the publish block without a second measurement.

What stays unproven is the third, and only the third: the upload itself has never run. No publish has gone through the shared block, so nothing has yet confirmed that the packaged .tgz still lands in the registry from there — and that is also the one of the three whose failure would be quiet rather than loud. The dispatch above is still the way to answer it.

The registry accepts a re-upload of an already published version (HTTP 201) and replaces the stored artifact. A republish of an existing version is therefore still silent.

The PR pipeline — validate-charts.yml

Eight jobs. Everything except the four render tests and the bump-level test follows the changed chart directories.

1. detect-changed-charts

Diffs the PR against its base and collects every changed top-level directory that contains a Chart.yaml. It emits three matrices:

  • charts — for the build job.
  • scan-charts — normally the same list. A PR that changes the scanner itself touches no chart, which would leave the change untested, so in that case a fixed sample is scanned instead: samba (a documented deviation), cyberchef (fully hardened) and matrix-synapse (two workloads).
  • install-charts — the changed charts minus the excluded ones.

2. build-chart (per changed chart)

ci/helm-dependencies.sh → helm lint --strict → helm package → version-bump check.

The bump check compares version: against the base branch and fails with a message naming the file. template-chart is exempt: it is never published, so its version is meaningless.

3. quantity-render (every chart, every PR)

The one job that deliberately ignores the changed-chart matrix. ci/run-quantity-render-test.sh renders every chart twice, injecting a bare integer memory request at every resources block the chart's values.schema.json declares:

Case Injected Expected limit Catches
bare-int requests.memory: 1048576, limitFactor: 2 2097152 the render aborting
fraction requests.memory: 999999, limitFactor: 1.5 1499998.5 a quantity in exponential notation

Both cases exist because the underlying bug has a loud and a silent half. Loud: the reformatted quantity carries the suffix e+06, the helper rejects it, helm template exits non-zero. Silent: the product of a fractional limitFactor and a bare number is reformatted after the suffix check, so the render succeeds and the manifest carries memory: "1.4999985e+06" — rejected by nothing until the API server sees it at apply time. Checking the exit code alone proves only half the case, so the second case searches the rendered text. 999999 stays below 1e6 on purpose, isolating the second formatting site.

Why all charts rather than the changed ones: the resources helper is copied into every chart's _helpers.tpl rather than shared through a library chart. The regression arrives by copy — a new chart started from an older one, or from template-chart before the fix — and in that PR the copied bug is in no diff anyone reads. Running over everything also removes a list nobody would maintain: adding a chart enrolls it automatically.

Two details worth knowing before touching it:

  • The resources paths come from values.schema.json, not values.yaml. It is JSON, so python3 reads it without pyyaml. A block counts when it is #/definitions/resources or carries limitFactor/requests itself, which keeps minecraft-server's JVM-sizing minecraft.resources out. A chart whose templates render a resources block while its schema declares none fails with exactly that message: the schema is out of sync.
  • A chart that never reaches the helper is reported, not counted as passing. gotenberg sets every computable limit explicitly, so nothing is multiplied and nothing is proven. Such charts appear under "helper not reached" in the job summary rather than as a green tick for a code path never entered.

4. selector-match (every chart, every PR)

The second job that ignores the changed-chart matrix, and for the same reason as quantity-render. ci/run-selector-match-test.sh renders every chart with the fixtures the install-test uses and checks that every label selector in the render really matches the labels of a pod template in that same render. Two kinds of object are covered, and both fail silently when their selector matches nothing:

Object What a non-matching selector does Who notices today
topologySpreadConstraints the constraint is a condition over zero pods and is therefore trivially satisfied: nothing is distributed, no event, no log line nobody — and with whenUnsatisfiable: DoNotSchedule the object reads like a guarantee
PodDisruptionBudget the budget does its arithmetic over zero pods and never blocks an eviction, so kubectl drain takes every replica at once nobody — kubectl get pdb shows an object that looks like protection

Neither is a render error, neither fails the install-test, and neither shows up in the diff of the PR that causes it — the chart that breaks it is usually a chart that renamed a label somewhere else. Charts answer half of that by keeping the selector and the pod labels in one helper (stalwart.groupLabels, keycloak's app label); the other half is that one definition only helps while nobody writes a second one, and that is the half a test can hold. Introduced with the per-group distribution rule in stalwart, where the trivially-satisfied case became reachable by design rather than by accident.

A spread constraint is checked against its own pod template (the failure guarded against is a constraint that does not count the pods it sits on); a PodDisruptionBudget against any pod template in the render, since it has no owning template.

Three things to know before touching it:

  • Helm is the YAML parser, through the one-template chart ci/yaml2json ({{ .Values | toJson }}). The runners have neither pyyaml nor yq — the same constraint ci/run-trivy-config.sh and ci/run-quantity-render-test.sh work under — and a YAML reader written in awk was the obvious alternative and the wrong one: its failure mode is a parser that quietly reads nothing, finds no selector, reports no mismatch and leaves the test green for a chart it never looked at. The rendered manifest stream is therefore handed to that chart as a values file and comes back as JSON for python3. The one textual step is the document split on a line that is exactly --- in column one, and getting that wrong does not produce a false pass: the fragment stops being valid YAML and helm exits 1. ci/yaml2json lives under ci/, so it has no publish workflow and no version-bump obligation — the chart-discovery loops look one level below the repo root and ci has no Chart.yaml.
  • What is deliberately not checked, named so the gap is not mistaken for coverage. podAffinity/podAntiAffinity are out of scope: their semantics are relative to pods that exist, so a term matching nothing is a legitimate configuration rather than a mistake. A selector expressed only through matchExpressions is reported as "not checked" and never counted as passing — evaluating set-based requirements means reimplementing label-selector semantics, i.e. a second parser by another name.
  • Required podAffinity is covered too, since Gitea #101. Preferred affinity and anti-affinity of either strength stay out of scope — their semantics are relative to pods that exist, so a term matching nothing is a legitimate configuration — but a requiredDuringSchedulingIgnoredDuringExecution podAffinity whose selector matches no pod in the release can never be satisfied, and the pod stays Pending for good. It is held against all pod templates of the render rather than its own, because selecting a different workload's pod is the whole point of such a term. First and so far only user: paperless-ngx' consume CronJob, pinned onto its StatefulSet pod's node; ci/selector-match/paperless-ngx.yaml is the overlay that makes it render at all.
  • Two anti-vacuum guards, because this test can be green over nothing. Both objects render only above a replica threshold and most charts' CI fixtures sit below it. ci/selector-match/<chart>.yaml is a render-only overlay applied last, whose only job is to push a chart over that threshold — and a chart that has one and still renders no selector fails, because that overlay existing is the whole claim that something was checked. On top of that, a run in which no selector at all was checked fails. Today stalwart is exercised through its overlay (two replicas in both node groups, plus an externalServices.valkey.host without which the chart's own "cluster with no in-memory store" abort fires) and keycloak through its ordinary install-test fixture, which already runs two replicas; paperless-ngx joins them through an overlay whose only job is to switch smbConnection on.

Measured: 30 charts, 5 selectors verified in ~14 s, no cluster. The fail path was exercised rather than assumed — renaming node-group to nodegroup in a hand-written copy of the PDB selector produced two FAIL lines naming both the selector and the labels actually present, plus a non-zero exit; the same was done for the spread constraint's selector, and the matchExpressions-only and overlay-without-selector paths were probed too.

5. httproute-match (every chart, every PR)

The third job that ignores the changed-chart matrix, and for the same reason as the two above: the mistake arrives by copy into a chart whose own diff nobody reads for it — and this convention is going to be copied into 20 more charts when the Gateway-API sweep runs (Gitea #114). ci/run-httproute-match-test.sh renders every chart with the fixtures the install-test uses and holds each HTTPRoute against the Service objects of that same render.

Finding What it does Who notices today
parentRefs empty the route attaches to no Gateway: created, schema-valid, rejected by nobody, routing nothing nobody — no event, no condition, the only symptom is a hostname that does not answer
backendRefs naming a Service, or a port, that does not exist the reference is resolved by name at runtime and validated by nothing nobody in CI — ResolvedRefs: False lands in a route status that only a Gateway controller writes, and none runs here
a prefix stripped with ReplacePrefixMatch while the same route carries no Exact match on that prefix with a RequestRedirect filter (Gitea #118) the slashless form of the path is rewritten to / and answered 200 by the app, under a document URL that still omits the prefix — so the page's relative links resolve against the host root, which on a shared host is a different application nobody — the app's own assets and XHRs are served by its neighbour, and the symptom is a half-working UI rather than an error

The real Gateway-API CRDs in the install-test close none of these gaps, which is the whole reason this job exists next to them: the Gateway-API schema accepts an empty parentRefs list, a backendRefs entry pointing at a Service that was never created, and a prefix-stripping rewrite with no redirect beside it. The CRDs prove the object is well-formed; this test proves it points somewhere, and somewhere right.

The third finding is a pairing within one HTTPRoute object, and the scope is deliberate: Gateway-API matching precedence is a property of a single route, so a redirect in a different route — a different hostname — says nothing about this one. Precedence also makes it a question of existence rather than of order, because an Exact path match outranks every prefix match: the redirect rule wins wherever it is emitted, and a chart's rule ordering stays a rendering convention.

freesailarr is the first chart it applies to — and, after group 3, still the only one. The line that used to stand here said bulwark was the second rewrite-target chart and would meet the same check when group 3 landed. Group 3 landed and it does not: bulwark has one Ingress object, no rewrite-target, and cannot have one — NEXT_PUBLIC_BASE_PATH is compiled into its bundle, so it cannot be served under a sub-path and its route is one rule at / with no URLRewrite filter. Every occurrence of the string rewrite-target in that chart is an explanation of why it does not help, and #114's measured table arrived at "bulwark, freesailarr" by grepping for the string rather than for the annotation. The script's own header carried the same claim and was corrected in the same round. The check still runs over every chart for the reason it always did — the mistake arrives by copy into a chart whose own diff nobody reads for it — not because a second chart is known to need it.

The port half is a genuinely new exposure rather than a hypothetical. An Ingress backend references its Service port by name; backendRefs takes a PortNumber and cannot. So from now on the number in the route and the port in the Service are kept in step by hand, and this test is what holds that pair.

Three things to know before touching it:

  • Helm is the YAML parser, through ci/yaml2json, for exactly the same reason as in selector-match above — no pyyaml, no yq, and a hand-written parser fails by reading nothing and reporting nothing.
  • What is deliberately not checked. That traffic flows — no Gateway controller runs in CI and none will, so Accepted: True is not obtainable and that claim is made nowhere in this pipeline. That any redirect is ever served — the pairing check proves the rendered object carries a RequestRedirect rule on the bare prefix and nothing beyond that: whether an implementation honours the statusCode or declines it and fails the whole route is outside what a render can see. Mutual exclusion with the Ingress — each chart aborts its own render when ingress.enabled and httpRoute.enabled are both true, which is where it belongs: this test renders one configuration per chart and a rule here would have to assume the switch is chart-global, which may not survive the multi-app charts of the sweep. Cross-namespace and non-Service backends, and a rule carrying no backendRefs at all (legal for a filter-only rule such as a redirect) — all three are reported as "not checked", never counted as passing. Note that the two interact honestly: the redirect rule the pairing check demands is itself backend-less, so a chart that satisfies the check also earns one "not checked" line, and both verdicts are correct at once.
  • Two anti-vacuum guards, because the convention ships httpRoute.enabled: false and this test would otherwise be green over nothing. ci/httproute/<chart>.yaml is a render-only overlay applied last whose one job is to put a chart into Gateway mode — and a chart that has one and still renders no route fails. On top of that, a run in which no assertion at all held fails. cyberchef is exercised through its ordinary install-test fixture (which installs in Gateway mode); every other converted chart through its overlay.
  • For a multi-app chart the overlay has a second job: it must enter every shape the template has, or the branch nobody exercised is the one that breaks. ci/httproute/freesailarr.yaml moves one app onto a host of its own on top of the fixture's shared host, so the shared-host route and a per-app route both render — and it leaves mediathekarr published-off, which exercises the per-app selector in the negative direction as well. It also keeps qBittorrent on the shared host, which is what makes the prefix/redirect pairing fire for a real chart rather than cover nothing.
  • And a multi-object overlay has to enter every object, which group 3 made routine: ci/httproute/keycloak.yaml covers the public and the admin route (the admin parent falling back to the chart-global one, which is the branch an installation takes), and ci/httproute/matrix-synapse.yaml covers the homeserver and the delegation route — two hostnames, two paths and two different backends, which is the only overlay in the repo where the two backendRefs resolve to different Services. Since #122 ci/httproute/keycloak.yaml also covers that chart's two realm routes, and it does so without repeating them: the two keycloak.realmHosts entries live in ci/keycloak/test-values.yaml and this overlay is applied on top of it, so one definition is installed as two Ingress objects by the install-test and rendered as two HTTPRoutes here. Defining them in the overlay instead would have left the Ingress half of that feature — the half carrying the required value and the convention deviation — with no coverage at all. The two entries are also the two Ingress forms, one with a clusterIssuer and one without, and what this test holds about those fields is their inertness: no tlsSecretName, clusterIssuer or per-entry annotation may appear anywhere in a rendered route. Those routes also bring a second rule shape — an Exact match carrying a RequestRedirect filter and no backendRefs at all, reported as "not checked" by design.
  • Two overlays have to supply hostnames, and for two different reasons. ci/httproute/homeassistant.yaml: that chart has no ingress.domain, so its hostnames come from ingress.hosts, whose default is the scaffold placeholder — one host with one root path. The overlay therefore sets two entries, the second with two paths including an Exact one, because that second entry is what proves the chart renders one object per host entry rather than one object with several hostnames. For that chart the overlay also adds coverage rather than trading it: ingress.enabled defaults to false there and the fixture does not switch it on, so the install-test had never exercised a routing object at all. Same situation as garage. ci/httproute/rss-bridge.yaml: that chart has a httpRoute.hostnames key, but it is mandatory with no fallback — no ingress.domain to fall back to, and it deliberately does not reuse ingress.hosts, whose entries carry tlsSecretName values that address nothing in Gateway mode. Its overlay sets three hostnames on one object, which is the shape its production installation has and the opposite of homeassistant's one-object-per-host. For that chart the overlay is the only automated coverage its Gateway mode has, because its install-test installs the Ingress mode — the ordinary situation across the sweep. See rss-bridge.
  • A render-failure path cannot live here, and two of group 3's guards are exactly that. This test only runs renders it expects to succeed, so homeassistant's pathType: ImplementationSpecific abort on a non-root path, and ytdlp-web-player's refusal to publish the app and its oauth2-proxy on one host, were exercised by hand instead. Where a guard matters, the chart's page says so and the PR records the exercise — a check that would have to assert a failing render belongs in the chart's own fail, not here.
  • What the overlays do not cover is the text of NOTES.txt. The Gateway branch of a chart's notes is executed by this test (a template error in it fails the job — verified by breaking one on purpose), but helm template does not print notes and the install-test installs in Ingress mode. So the notes' Gateway wording is reviewed by reading, never by a check.

Measured in CI (run 719, the pilot): 30 charts, 4 assertions held, whole job 13 s including checkout and the Helm setup, no cluster. Group 1 of the sweep took that to 20 assertions over 10 charts and group 2 to 36 over 15 — the sweep adds charts, not checks. The prefix/redirect pairing of Gitea #118 is the first check added since the pilot rather than a chart, which brought it to 37 over 15: one assertion for freesailarr's /qbittorrent, plus one further "not checked" line for the backend-less redirect rule it demands.

Group 3 took it to 56 assertions over 21 charts. Its 19 come from six charts, and three of them contribute more than one route: keycloak 4 (public + admin), matrix-synapse 4 (homeserver + delegation, two different backends), homeassistant 5 (two routes, three rules), bulwark 2, stalwart 2, ytdlp-web-player 2.

rss-bridge finishes the sweep at 58 assertions over 22 charts — one route, so two assertions. It was the last chart left: 22 charts carried an Ingress template and 21 an HTTPRoute template, and the difference was exactly that one, held back by Gitea #51 rather than by anything about its shape.

The realm hostnames of Gitea #122 add 4, and they are objects rather than a new check: keycloak's share goes from 4 to 8 — one parentRefs and one backendRefs assertion for each of its two realm routes — plus one "not checked" line each for their backend-less redirect rule (keycloak: ok (8 assertion(s) held, 2 not checked)). The #118 prefix/redirect pairing deliberately does not fire on them: it is scoped to rules that strip a prefix with a ReplacePrefixMatch rewrite, and a realm route strips nothing — it redirects the bare root of a hostname that serves one realm. The two rounds were measured separately and neither total includes the other: 58/22 was read on the rss-bridge branch and 60/21 on the realm-hostname branch, which did not contain rss-bridge. So the delta is the figure to trust from either, and and the running total was read off the first branch that held both — the realm-hostname branch with main merged into it — rather than added up by hand: 62 assertions over 22 charts, 0 failed. keycloak contributes 8, one parentRefs and one backendRefs assertion for each of its four routes.

That chart's install-test also went green in the same PR, after failing on this runner since the move — the event #51 had designated as its own signal, and the reason the chart was deliberately never added to ci/excluded-charts.txt. Pod 1/1 Ready in 22 s, 0 restarts, no Permission denied in the job. The chart is ruled out as the cause by measurement rather than by argument: that PR's rendered defaults are byte-identical to origin/main, so the pod spec that installed is the one that had been failing. The runner is the implicated layer. It reproduced on every run of that branch after the first, so it is not a single fluke, and #51 was closed on that round. The cause, however, was never identified — nobody established what changed on the runner, and that issue's diagnostic programme was not executed. A fresh red on this chart is therefore a finding, not a known quirk. See rss-bridge.

The fail path was exercised rather than assumed: a Service renamed in the route, a port the Service does not expose, a port left out entirely, an emptied parentRefs with the chart's own guard disabled, an overlay that renders no route, a run with zero assertions, and — for the pairing — both the redirect rule deleted outright and the redirect pointed at a path other than the one stripped; each produced a FAIL naming both sides and a non-zero exit.

The MTA-STS routes of Gitea #123 add 2, the third of the three parallel rounds and the one that shows what this figure means from now on: stalwart goes from 2 assertions to 4 — one parentRefs and one backendRefs assertion for its new MTA-STS route — and the chart count does not move at all. Every earlier entry in this history grew because a chart joined the list; this is the first that grew because a chart already in it gained an object. That is the ordinary shape of the next one too, now the sweep is done.

And this is where the delta rule gets cashed in: 64 assertions over 22 charts, 0 failed. The three rounds ran in parallel on branches that contained none of each other, so each published a delta — +2 for rss-bridge, +4 for the realm hostnames, +2 here — and each deferred the running total to "the first run on main that holds all of them". This round was the last of the three to merge, so its own run is that run: rebased onto a main already carrying the other two, ci/run-httproute-match-test.sh reads 64 over 22, with stalwart contributing 4.

Worth keeping the intermediate figures legible rather than tidying them away, because each was correct for what it was measured on and none of them is the total: 58/22 on the rss-bridge branch, 60/21 on the realm-hostname branch, 60/22 on this one before #122 landed. 62 + 2 = 64 happens to agree, and that agreement is a check, not the source — the number above was read, and the arithmetic is only reassuring because nothing else moved in between. Where a round changes an existing chart's routes rather than adding a chart, the deltas would not add up like that at all.

And it only counts because the overlay was extended to enter the new shape (ci/httproute/stalwart.yaml carries an entry with a clusterIssuer, one without, and one host on a subdomain). An overlay left alone would have added a route this test never rendered — the anti-vacuum rule biting on a second object inside a chart that already passes, which is a gap no per-chart "has an overlay" check can see.

Note which half that leaves uncovered: an overlay switches a chart into Gateway mode, so the Ingress half of a both-modes feature is never rendered here. For stalwart's MTA-STS objects that half sits in ci/stalwart/test-values.yaml instead, where the install-test has a real API server accept them.

6. bump-level (the version-bump hook, every PR)

The fourth job that ignores the changed-chart matrix, and the only one that renders no chart at all: ci/run-bump-level-test.sh tests ci/renovate-bump-chart.sh, the postUpgradeTasks hook that computes a chart's version: after Renovate moved an app version. Added with Gitea #128, which is what made it necessary — until then the hook was a calculator with one branch (app major → chart major, everything else → patch); with the bump level taken from Renovate's update type it became the thing that states how big a change a cluster is being offered, and the chart version is the only signal anything downstream sees.

Its failure mode is the silent kind: a wrong level is a valid version string. The PR goes green, the chart publishes, and the only symptom is a number that understates what it ships. Nothing else in the pipeline reads that number, so nothing but a test can.

  • A case is a repository, not a function call. The script reads the base commit (git show HEAD:<chart>/Chart.yaml) and the working tree, so each of the 49 cases gets a throwaway git init under mktemp -d, one commit with the base Chart.yaml, and the new one written over it uncommitted — the state the hook really runs in. Signing is switched off per command, because this workspace signs through an SSH signer and a throwaway repo has no business invoking it.
  • No Helm, no cluster, no Python, no Docker. The fixtures are six-line Chart.yaml files written with printf and the assertion is one version: line read back with sed, so the job carries no Helm step and takes about a second. Same runner constraints as the render tests (no pyyaml, no yq) — simply not needed here.
  • What is covered: each update type; the freesailarr shape (an app minor while appVersion stands still); all seven non-semver app versions plus the majors that must not fall back; the maximum rule in both directions, including a stale branch version that must not be kept; a missing, an empty and an unknown second argument; and the inputs the script must leave alone (a chart new on the branch, template-chart, a .github/ or ci/ argument, no argument at all). The full version: line is asserted every time, so the +up suffix and its dash-to-dot normalization are covered throughout.
  • What it cannot assert, named so the gap is not mistaken for coverage: that Renovate passes the update type at all. The hook runs only where the command is allow-listed, which is on the shared instance and outside this repository — which is exactly why a missing second argument is one of the cases. See Renovate → the version-bump hook.

7. yaml-scalar (every chart, every PR)

The fifth job that ignores the changed-chart matrix and the fourth that renders every chart (Gitea #11). ci/run-yaml-scalar-render-test.sh renders every chart with the fixtures the install-test uses and checks that every scalar Kubernetes types as a string really arrived as one. Three assertions, each a property the API itself requires — so there is no chart for which the opposite is a legitimate configuration, and none of them needs a per-chart expectation:

Assertion A bare render does this Who notices today
CronJob.spec.schedule is a non-empty string a schedule beginning with * or @ aborts the render — YAML reads the leading character as an alias or a reserved indicator the operator, the hard way: the error names the template, their own value appears nowhere in it, and quoting it in the values file does not help, because Helm passes the string's content through
every env[].value of every container and initContainer is a string "true" and "0755" are legal plain scalars that are not strings — the chart renders perfectly and the API server rejects the object with cannot unmarshal bool into Go struct field EnvVar.value of type string nobody until apply time, which here means the install-test or a cluster, never the render
every metadata.labels and metadata.annotations value is a string both are typed map[string]string, so a unit-less 100 in an upload-size annotation is rejected whole nobody in the render

The defect has a loud and a silent half, like quantity-render's, and the same consequence: checking the exit code alone covers one of them. The second and third assertions read the rendered output back and type-check it.

  • Why this job and not a fixture. The object that carried the original defect was rendered by nothing. paperless-ngx' consume CronJob sits behind smbConnection.enabled, false in the chart defaults and in its install-test fixture, deliberately — the CI cluster has no SMB CSI driver. So neither lint, nor the other render tests, nor the install-test ever produced that template's output, which is how an unquoted schedule shipped for months. A test over what the fixtures already render would have been just as blind.
  • Two anti-vacuum guards, and here the overlay is the test. ci/yaml-scalar/<chart>.yaml is a render-only overlay applied last, whose job is to switch such objects on and to feed them hostile values — a star-leading schedule, an env value that is a bare true. A chart that has one and still renders none of the three kinds fails, and a run that asserted nothing at all fails. Today only paperless-ngx has one.
  • Helm is the YAML parser, through ci/yaml2json, for exactly the same reason as in selector-match — no pyyaml, no yq, and a hand-written parser fails by reading nothing and reporting nothing.
  • What is deliberately not checked. A value rendered into a scalar that is hand-quoted in the template ("{{ .Values.x }}") is a string in the output by construction, so this test is blind to it — what it can still do is break the YAML when the value itself contains a quote or a backslash, and only | quote fixes that. Everything that is not a label, an annotation, an env value or a schedule: a bare storage: or className: is equally unquoted, but its value space is DNS labels and quantities, where no legal value is YAML-ambiguous. And whether a cron schedule is a valid schedule — the controller's judgement; narrowing the schema to a regex was considered and rejected, since the defect was never a wrong schedule but a correct one rendered bare.

Measured: 30 charts, 408 scalars, 0 failed, no cluster. The fail path was exercised in all three shapes against a throwaway copy of the chart as it stood on main — the bare schedule aborting the render, the star-valued env entry aborting it on the StatefulSet once the schedule was quoted, and finally the bool-looking value on a render that succeeds, where the type assertion named the variable, the container and the API server's own message. Six charts still render their env block bare (matrix-synapse, minecraft-router, minecraft-server, rallly, satisfactory, teamspeak-server): they pass only because no fixture of theirs sets a hostile value, and bringing one over is a major bump for that chart.

8. networkpolicy (every chart, every PR)

The sixth job that ignores the changed-chart matrix and the fifth that renders every chart (Gitea #5). ci/run-networkpolicy-test.sh holds every NetworkPolicy a chart renders against the pod specs, probes and Services of the same render. It exists because #5 put a policy into all 29 installable charts at once, on by default — 29 objects whose every failure mode is silent, since the API server accepts a policy that filters nothing exactly as readily as one that filters everything.

Assertion What goes wrong without it Who notices today
the podSelector matches a pod spec of the same render the object sits in the namespace, is listed by kubectl, reads as a boundary and filters nothing — usually because a label was renamed in the workload template, a different file and a different diff nobody, ever: nothing is denied, so there is no symptom to trace back
every ingress port is a containerPort of the selected pods a NodePort number where a pod port belongs. The policy is evaluated at the pod's interface, after kube-proxy translated the port, so 30025 matches nothing while looking strict — and being the only rule for that port, it blocks every inbound connection instead nobody in CI (see below), and in production as a service that went mute
no ingress rule allows every source by accident from: [] allows everything, identically to no from at all, while reading in a diff and in kubectl output as a restriction — the one mistake here that makes a policy weaker while looking stricter nobody
every port a network probe addresses is either unrestricted, or the chart offers probeSources and says in NOTES.txt what a wrong CIDR costs the kubelet probes from the worker node's IP, which no chart can know. A restricted probe port without it drops every probe: the pod never becomes Ready and the symptom reads as a broken application the install-test, under production's CNI — but not in CI, see below
cross-chart egress (#136): where a chart dials another chart of this repository, its egress rule must contain the pod ports behind that chart's Services an egress rule naming the Service port. kube-proxy does the ClusterIP's destination NAT before the policy is evaluated at the pod's interface, so the policy sees podIP:targetPort and the rule matches nothing wherever port and targetPort differ nobody: the symptom is a timeout, not connection refused, and no error names a NetworkPolicy. paperless-ngx 14.0.0 shipped it and lost Tika and Gotenberg in production on 2026-10-15

Egress ports are otherwise deliberately not held against the pod's containerPorts: an egress port is a port on the destination, and that comparison would fail every chart that reaches a database.

The fifth row is the exception, and the only assertion here that needs two charts. For a database, a cache or the web the destination is in no render this repository has — but it is in a render when one chart dials the Service of another, and paperless-ngx → apache-tika, gotenberg is the single place where that happens (checked for #136: the three other policies naming port: 80 aim at the internet, where there is no DNAT and 80 is the real port). The pairs are an explicit table in the script (EGRESS_PEER_ROWS), not a discovery, because no render can know which Service another chart dials — the address is a value or a 1Password field — and a row naming a chart that no longer exists fails rather than being skipped. Each row runs against two renders of the consumer: with the feature gate on the containment must hold, resolved Service port -> targetPort -> containerPort so the evidence line prints the whole DNAT chain (port 80 -> targetPort 'tika-port' -> pod port 9998); with the gate off the requirement must not apply and those ports must be gone, so a gated rule cannot quietly become one that permits what the release cannot use. It lives in this script rather than a sixth one because the job already renders every chart on every PR, Helm is already the parser, and a second script would duplicate all of that to add one comparison.

  • Two guards beyond the four, both earned. A chart that renders no policy fails — that is how this issue decayed in the first place, since garage and keycloak were created after it was filed and arrived without one. And a # comment inside a rendered policy fails, because a # comment is part of the object: it ships to the cluster, appears in helm get manifest, and lands in every release diff. Both precedent charts failed that check the first time it ran, which is what it is for.
  • Assertion 3 is stricter than it reads. A missing from is the documented open form and both precedent charts use it on purpose, so an outright ban was not available. It is accepted only where the render itself justifies it: every port of the rule is probed, or is published by a NodePort Service in the same render. Anything else is an allow-all nobody chose.
  • Subchart documents are skipped by the comment assertion, structurally rather than leniently: a dependency is fetched from the registry, not from the copy in this repo, so the policy paperless-ngx and stalwart bundle is whatever valkey version is published — and nothing in the parent can change its rendered comments. The subchart's own PR catches it, because this job runs over valkey too.
  • Anti-vacuum guard. ci/networkpolicy/<chart>.yaml is a render-only overlay applied last, whose job is to render the source-restricted probe rule — a branch no chart's own defaults reach, and the one holding the expensive mistake: a namespaceSelector and a podSelector as two list entries instead of one would be an OR, opening the port to every pod in kube-system. A chart with an overlay that renders no restricted probe rule fails. The counter is specifically the probe rule, because keycloak restricts two rules in its own defaults and would have satisfied a looser guard while the branch went unrendered.
  • Helm is the YAML parser, through ci/yaml2json, for the same reason as the render tests above.

Measured: 30 charts, 210 assertions held, 0 failed, no cluster, ~15 seconds. All twelve fail paths were exercised rather than assumed, and one earned its keep immediately: the probeSources check was a substring test, so a chart whose key had been deleted while the comment above it still discussed the value passed. It now looks for the key.

The cross-chart assertion is two of those 210, and its own six fail paths were executed the same way: a required port removed from the rule, a wrong number in its place, the peer chart renamed away, the policy's name no longer following the <release>-<chart>-<kind> convention, the peers rendering no Service at all (an empty requirement reads exactly like a satisfied one, so it fails), and the gated rule made unconditional — which is the fail path of the gate-off half, since with the gate off and the rule properly gated the assertion must stay silent.

What no part of this pipeline covers, and it was measured rather than assumed. On the PR for #5 two experiments ran and were then reverted:

Experiment Result What it means
cyberchef's probe port restricted to 192.0.2.11/32, an address no probe can come from install-test green, 1m9s k3s exempts the kubelet's probe traffic from policy. A blocked probe path does not turn the install-test red
outline's DNS egress rule removed, so postgres and redis do not resolve and /_health cannot pass install-test red, 5m59s policies are enforced in CI. The net is real

So the install-test's net is an egress net — it catches anything the application itself needs, not a blocked probe path. Two consequences worth keeping: the probe-path trap is not reachable from the defaults of this convention (the rule is open until an operator sets probeSources, which is what the fourth assertion and the NOTES.txt sentence are for); and a wrong ingress port is caught by assertion 2 or not at all, since probes are exempt and no ingress controller runs in CI either. That the path from Traefik to a pod works is proven nowhere in this pipeline.

9. trivy-config (reporting only)

Renders each chart in the scan matrix and lists what trivy config finds. It never fails the job (TRIVY_CONFIG_MODE: report). Deliberately not a gate yet: the repo has several documented deviations, and a check that goes red on those every time is one people learn to scroll past. Switching the mode to gate is the intended step once the seccomp gap is closed.

Trivy is installed from its release tarball, not via aquasecurity/setup-trivy — see below.

Exceptions live in ci/trivy/trivyignore.yaml, each with a mandatory statement — the script refuses to run without one and prints every suppressed finding together with its reason. The trusted-registry list for AVD-KSV-0125 is a separate file on purpose, ci/trivy/data/ksv0125.yaml: it is configured rather than suppressed, so the check keeps being able to say something about an image from a genuinely foreign source. A new registry therefore means an entry there, not an ignore — quay.io joined it with keycloak, the first chart in the repo using it. Note what the check does not forgive: an unqualified Docker Hub image puts the organisation in the registry position and is still reported, which is why every chart writes its image fully qualified.

10. install-test (per changed, non-excluded chart)

ci/run-install-test.sh <chart> against a throwaway k3d cluster: apply the stub OnePasswordItem CRD plus the chart's fixtures, helm install with the CI values, and wait for every Deployment and StatefulSet to roll out. Since probes and hardening are active, "Ready" means the app actually answers. On failure the job dumps pod descriptions, events and container logs.

The job does not live in validate-charts.yml. It is the reusable workflow install-test.yml (workflow_call, one input: a JSON array of chart names), called from here with the changed charts and from install-test-all-charts.yml with all of them. Everything that decides how a chart is installed is in that one file — the runner-label expression, the pinned kubectl with its skew check, envsubst, the containerised-job detection, the per-leg cluster name, the freesailarr credentials; the callers decide only which charts and whether the job runs at all. It is not copied for the second caller: a second copy is a second chance to test differently, unnoticed, which is the same argument that gives ci/excluded-charts.txt exactly one parser.

Every leg records its own outcome in the job summary next to the four facts that decide what that outcome means — date, runner label, K3S_IMAGE, kubectl version — and it distinguishes "the chart did not come up" from "the cluster never came up", because the two mean opposite things.

The job blocks the PR. That stopped being negotiable when Renovate began automerging patch and minor updates: a chart that no longer starts must not reach main and publish unattended. It counts double now that automerge covers every update type — validate-charts is a required check on main, so a red install-test keeps the armed PR where it is.

Install-test fixtures

Fixtures live at the repo root under ci/, deliberately not inside the chart directories — so they are neither packaged into the chart .tgz nor do they trigger the publish workflows on merge.

Path Purpose
ci/common/ applied for every chart (client-side): the schema-less stub CRD for OnePasswordItem
ci/gateway-api-crds.sh the real Gateway-API CRDs, applied --server-side and only for a chart whose render actually contains an HTTPRoute — see below
ci/httproute/<chart>.yaml render-only overlay for the httproute-match job, never used at install time
ci/yaml-scalar/<chart>.yaml render-only overlay for the yaml-scalar job: switches objects on that no fixture renders and feeds them YAML-hostile values, never used at install time
ci/selector-match/<chart>.yaml render-only overlay for the selector-match job: pushes a chart over the threshold at which it renders a selector, never used at install time
ci/<chart>/test-values.yaml values for the CI install — required for every installable chart
ci/<chart>/fixtures.yaml dummy secrets, throwaway infrastructure (postgres/redis/dex), static hostPath PVs and storage classes, seed jobs
ci/<chart>/fixtures.env.yaml fixtures needing a value that must not be committed: ${VARIABLE} placeholders only, rendered with envsubst and piped straight into kubectl, never printed
ci/<chart>/settings.env INSTALL_TIMEOUT for slow starters, REQUIRED_ENV="VAR_A VAR_B" declaring what fixtures.env.yaml needs
ci/excluded-charts.txt <chart> <justification>, one per line

Substitution is restricted to the names in REQUIRED_ENV, so an undeclared variable is never filled in and no unrelated $… in the manifest is touched. A missing or empty declared variable fails the install-test loudly, naming the variable — it never skips the chart. Skipping would restore exactly the green-but-meaningless signal the mechanism exists to remove; the one legitimate case of absent secrets, a PR from a fork, belongs in an if on the job.

freesailarr is the only chart using fixtures.env.yaml: gluetun needs a real Wireguard key, or it never completes a handshake and no app container behind its kill switch becomes Ready. The key is loaded from 1Password for that chart only, with :? guards that name a missing op:// field before the cluster is even created. It is a CI-only Mullvad key, deliberately not the production one: Mullvad ties an address to a key and routes the tunnel to the endpoint it saw last, so a shared key would make every PR run pull the tunnel out from under the running stack — symptom in the cluster, cause in a pull request.

Fixture images avoid Docker Hub where possible (public.ecr.aws, ghcr.io) to dodge runner rate limits.

The Gateway-API CRDs are real, pinned and applied server-side

Added with the Gateway-API pilot (Gitea #114). ci/gateway-api-crds.sh fetches the standard channel at a pinned version — tracked by Renovate through a customManager for ci/*.sh, the same # renovate: comment convention the workflows use for Trivy and kubectl — and applies it. ci/run-install-test.sh calls it only when the chart's render actually contains an HTTPRoute, detected from the render rather than declared per chart, so there is no list to maintain and the sweep enrols a chart simply by switching its fixture into Gateway mode. A false negative there is loud, not silent: without the CRDs helm install fails with no matches for kind HTTPRoute in version gateway.networking.k8s.io/v1.

Three decisions, each with the measurement behind it:

  • Not a stub, unlike OnePasswordItem. A schema-less stub is right for that CR — nobody is testing the 1Password operator's API, only that the chart's object applies. For an HTTPRoute the schema is the entire point of the exercise: with a stub the API server accepts a malformed parentRefs, a port written as a string, a path type that does not exist, the install goes green, and the only thing demonstrated is "something was created". With the real CRDs the same apply is validated field by field.
  • kubectl apply --server-side, and not out of taste. Client-side apply stores the whole applied object in the kubectl.kubernetes.io/last-applied-configuration annotation, and an object's annotations may total at most 262144 bytes. The HTTPRoute CRD of the standard channel is 429367 bytes of YAML, which comes to 244792 bytes as JSON (measured on v1.6.3, re-read through ci/yaml2json) — 93 % of the limit, with every upstream release adding fields. Client-side apply therefore happens to fit today and is one release away from failing with metadata.annotations: Too long on a Renovate PR that has nothing to do with it. Server-side apply writes no such annotation and has no such limit.
  • Not in ci/common/, which is applied client-side for every chart: putting the CRDs there would hand that landmine to all twenty-odd charts and spend the download on charts that render no route at all.

It waits for condition=Established rather than for the objects to exist. kubectl apply returns as soon as the CRD object is created, while the API server needs a moment to serve the new endpoint — and a helm install in that window fails with no matches for kind HTTPRoute, which reads like a missing CRD rather than like a race. Same lesson as the Ready-node loop in install-test.yml: wait for the condition that means usable, not for the one that means present.

Downloaded rather than vendored, like Trivy's tarball, k3d's install script and kubectl. 1.1 MB of generated YAML in the repository would go stale invisibly and would bury every diff touching ci/. The price is named instead of hidden: one more network fetch per affected leg, and if the release asset ever disappears an unrelated pull request goes red — loudly, which is the acceptable half.

What this still does not prove: that traffic flows. No Gateway controller runs in k3d, so no route is ever reconciled and no status is ever written. Provable here: the object is schema-valid and the pod becomes Ready.

Excluded charts are dropped from the matrix, not skipped inside it

An excluded chart used to get a job that ran the script, printed SKIP and exited 0 — and the forge knows only the exit code, so the check list showed Install-test satisfactory ✓ for a chart whose image was never pulled, indistinguishable from one that really came up. detect-changed-charts now removes the excluded charts before the matrix is built, so no job exists and nothing claims they passed. The all-charts run does the same in its select-charts job.

Because an absence on its own says nothing, the job that builds the matrix writes every excluded chart with its documented justification into the job summary. The list is parsed in exactly one place: ci/run-install-test.sh --list-excluded emits <chart><TAB><reason> and both workflows call it rather than re-implementing the format in YAML — a second parser would be a second chance to select different charts, silently and in either direction. The script keeps its SKIP branch for local runs.

What cannot be done instead: deciding per chart in an if on the job, to get a visible Skipped rather than an absent entry. It is rejected with context "matrix" is not allowed here — jobs.<id>.if is evaluated before the matrix is expanded, so the matrix context does not exist there, and the whole workflow ends in failure with zero jobs created.

The justification is mandatory

A line in ci/excluded-charts.txt carrying only a chart name is deliberately not an exclusion: a bare name would drop a chart out of CI with nothing to review.

The all-charts install-test (on demand)

install-test-all-charts.yml runs the same job as the PR pipeline — the same install-test.yml, the same pinned kubectl, the same fixtures — with a matrix over every installable chart instead of the ones a pull request changed. workflow_dispatch only, no schedule.

Why it exists

The install-test matrix follows the charts a pull request changes. So there is no moment at which the charts are checked as a whole against the current environment — and a chart nobody has touched since the forge move carries a green history from a different runner. That green says nothing about today, which is exactly the signal rule 2 is about.

The value of the answer was demonstrated by accident. Gitea PR #48 corrected a comment in 21 charts, which made every one of them "changed" and turned the run into the first all-charts install-test after the move: 19 of 20 matrix charts installed and became Ready in 12m41s, in one job, on the Gitea runner with the pinned kubectl inside the job container — and exactly one did not. That one was rss-bridge, whose last green run at the time was from 2026-08-29 and a GitHub runner (Gitea #51). Without the comment round, nobody would have learned either half. That chart has since gone green on the Gitea runner too — on 2026-10-13, in a PR whose rendered defaults were byte-identical to main, which is what rules the chart out and leaves the runner.

When to run it

  • after a change to the runner
  • after a change to K3S_IMAGE or the pinned KUBECTL_VERSION
  • after a change to the fixtures under ci/
  • after a forge move
  • and for a pull request that only changes the harness and therefore tests nothing by itself — run it once the change is on main

Not on a schedule: it would spend compute every day on a question that changes rarely, and a red run nobody asked for is a red run nobody reads.

How it behaves
Input charts space- or comma-separated chart names, empty = all. This is what lets a single chart be tested against the runner, which is what Gitea #51 needs.
Unknown or excluded name a loud error, not a silent omission — someone typed that name, and a green run over zero charts would be the worst possible answer. An empty matrix fails the run for the same reason.
One cluster per chart no reuse. Reuse saves time and changes what the test proves: leftovers of the previous install, a warm image cache, fixtures already applied. "Came up from nothing" is the statement.
Excluded charts dropped through the one parser, ci/run-install-test.sh --list-excluded, and listed in the summary with their justification, as in the PR pipeline.
The record per chart: outcome, date, runner label, K3S_IMAGE, kubectl version. Plus a final job with the verdict over the whole matrix. A green run nobody notes is unknown again next time.
Expected runtime 20 charts in 12m41s (Gitea PR #48, one job, all legs concurrent on one Docker daemon). Written down so a later deviation is visible as one; compare it against the chart count the run prints in its own summary — 24 at the time of writing, and one more with every new chart. The timeout is 30 minutes per leg, in install-test.yml, hence identical for both callers. A big image moves this less than it looks: convertx is ~1.4 GiB compressed, by a wide margin the largest in the repo, and its first green leg measured 2m35s including cluster creation (Gitea PR #61) — inside the existing envelope, not a new one. The legs run concurrently, so this figure tracks the slowest leg plus scheduling, never the sum of the pulls.

SonarQube analysis

sonar.yml is a hull of the same kind as the publish ones: it decides when, and the workflow itself is the shared block sonar-analysis in workflow-util/pipelines, bound to a tag. The project-specific half is sonar-project.properties in the repo root. Why fetch-depth: 0 is not optional, why the block needs no Docker runner, and what the scanner downloads at run time are on that page and deliberately not repeated here.

on:
  push:
    branches: [main]
  workflow_dispatch:
jobs:
  sonar:
    uses: workflow-util/pipelines/.gitea/workflows/sonar-analysis.yml@v4
    secrets: inherit

It observes, it does not gate

The Community Build analyses the main branch only — there is no pull-request analysis to hook into. So the trigger is a push to main, and a finding appears after the merge. At that point there is nothing left to block, which is why the block carries no quality-gate step.

That is not a gap waiting to be filled. A red gate behind the merge would colour main red without having prevented anything, and a permanent red is a thing people learn to ignore — at which point it has spent the credibility of validate-charts, the one check here that can stop a merge. It becomes the right thing under one of two conditions, and both change the situation rather than the wiring: pull-request analysis is available, or a deployment hangs off the workflow.

Practical consequence for whoever reads a chart PR: the analysis will never be in its check list. The findings are in SonarQube, about main, and they are read there.

What is analysed

Measured in run 375, not read off a language list. The earlier version of this section was read off a language list and promised more than the instance delivers — the correction below is the whole point of the section now.

The server loaded exactly two quality profiles, json and yaml. The sensors that did work were the IaC Kubernetes sensor (3.4 s) and the secrets detection (TextAndSecretsSensor) over all 391 indexed files.

What is in this repository Analyser Runs here?
238 YAML files and 26 Helm template directories IaC Kubernetes, YAML yes
26 values.schema.json JSON yes
the whole tree, 391 indexed files Secrets yes
the 5 shell scripts under ci/ — the whole CI logic Shell no
the 30 workflow files GitHub Actions no

Why the last two do not run, and why no change here can make them: analysers are server-side plugins. The scanner downloads a CLI, not a set of languages; what gets analysed is decided by what the SonarQube instance has installed. The language list that promised Shell and GitHub Actions describes the newest SonarQube release, and this instance does not carry those plugins. Nothing in sonar-project.properties and nothing in the shared block changes that.

The consequence, said plainly because it inverts the original expectation: the files that carry the actual logic of this repository — the five shell scripts under ci/ and the thirty workflow files — are not analysed at all. What is analysed is the declarative part: IaC rules over the charts, and secret detection over everything. For the scripts, shellcheck and review remain the only check, as they were before Sonar existed here.

Whether the instance is upgraded or has those plugins added is a decision for whoever operates SonarQube, not something this repository settles. Noted here so the gap is known, not as a task on this side.

The secrets row is the reason to bother, and it does run. The premise of this repository is that a credential never lands in it — every secret is a 1Password OnePasswordItem reference, materialised by the operator in the cluster — and until now nothing checked that premise. The two surviving rows above it are a second opinion on something helm lint --strict, the render test or Trivy already touch; this one covers an assumption carried for two years on discipline alone.

The analysis itself works. Run 399, the push that merged the corrected project key, finished successful in 44 seconds and reported ANALYSIS SUCCESSFUL, you can find the results at: https://sonar.jk-dev.de/dashboard?id=JLab-Helm-Charts. Sensors that did work: IaC Kubernetes, TextAndSecretsSensor, the JavaScript/TypeScript/CSS sensor (which finds nothing here) and the IaC project sensor. Shell and GitHub Actions remain absent, as in run 375.

What it found is not readable from the pipeline. The findings live on the server, and that server does not answer anonymously: api/measures/component replies 401 while the dashboard page itself returns 200. That is the right setting — an instance that hands out analysis results to anyone would be a finding in its own right — but it means the numbers have to come from whoever has an account there, not from a log. Whether the Helm templates produce noise, and whether the secrets detection finds anything in a repository whose whole premise is that it holds no credential, can only be read off that dashboard.

No exclusions, for the first round

Including the Helm templates, which are not valid YAML because of {{ }}.

What the Helm and YAML analysers make of such a file is only learnable by letting them run over one. Excluding them up front would discard that answer and, with it, the bulk of the repository — and it would be an exclusion written on a guess, which hides whatever else happens to match it. There is no paths filter on the hull either, for the matching reason: a filter would leave the previous result standing after a change it did not cover, while still presenting it as current.

If the first run produces noise genuinely not worth reading, the exclusions go into sonar-project.properties with the finding they answer named next to them. That file is ours; the shared block never carries an exclusion for one repository.

Prerequisites, and what a missing one looks like

Name Art
SONAR_HOST_URL repository variable Set on the instance. A variable and not a secret on purpose: a server address is not a credential, and as a variable it stays readable in the log — which is the only thing that makes a wrong address diagnosable. Masked, it would produce a failure against an address nobody can see, the exact situation that forced an extra masking step into the publish block.
SONAR_TOKEN repository secret Confirmed present by the maintainer, and it reached the scanner in run 375. Reaches the block through secrets: inherit. Needs the "Execute Analysis" permission on the project in SonarQube.
sonar-project.properties file in the repo root Project key JLab-Helm-Charts, and no sonar.projectName. See below — both halves of that line were learned the hard way.

Secrets and variables are not readable from outside, including by the bot: tea actions secrets ls answers user should be the owner of the repo, and variables are not listed at all. So "set" here means confirmed by the maintainer, not verified by a tool — and the block's guard is what turns a wrong assumption into a named error instead of a puzzling failure.

The guard covers an empty variable, an empty secret and a forgotten secrets: inherit, and it checks both values before it exits, so a repository that has set up neither learns both in one run. What it cannot cover is a missing required secret: that fails job creation on this instance, not the job, and the run then shows no steps at all — the same quirk as everywhere else here, and the understandable case for this block, because without a token there is no analysis to run.

The project key, and what the first run cost

Project keys are not derivable. The key here is JLab-Helm-Charts, because that is what the project already has on the SonarQube instance; it was given by the maintainer and read off the server. The first version of this file carried a guessed key, jlab-cluster_helm-charts, reasoned from the instance serving two owners — plausible, and wrong, and the run was red for that one reason:

ERROR You're not authorized to analyze this project or the project doesn't
      exist on SonarQube and you're not authorized to create it.

That message does not distinguish a wrong key from a missing permission — the server answers the same way for both, which is precisely what made a guessed key expensive to diagnose. Now that the key is known to be right, the same error means the other half: the token lacks "Execute Analysis" on that project.

And no sonar.projectName. Once the project exists on the server, the display name belongs to the server. A name sent along with the analysis overwrites it on every run — so a rename made in SonarQube would silently snap back, with no failure anywhere to report it. The reason sits as a comment in sonar-project.properties, because otherwise the next person adds the line back as an obvious omission.

Three things the first run answered — all three positively

Run 375, a push to main. It was red at the upload, but everything before the upload ran, which is what makes it a measurement.

  1. Is node24 in the runner image? Yes — the scan action is a JavaScript action declaring runs: using: node24, and it ran. Had it been absent, the failure would have read as a problem with the action rather than with the image, which is why this was written down in advance.
  2. Does the scanner download get through? Yes — the archive and its GPG signature, ~59 MiB, in roughly three seconds. This was the expected first failure, and it had history: a 17 MB download once failed on this runner because a Docker bridge at MTU 1500 met a pod network of 1450 (see What a GitHub runner had been providing quietly). The node-side fix holds at better than three times that size; the MTU incident does not repeat.
  3. Does a repository variable reach a workflow called from another repository? Yes. The guard passed and the scanner started, which it could not have done against an empty vars.SONAR_HOST_URL — the block checks the variable and exits first. This also answers it for OP_CONNECT_HOST in the publish block: same forge version, same mechanism, a repository variable read inside a workflow called from another repository. No second measurement needed there, and the contingency plan — passing the address in as an input, as was done for the owner segment — is not needed either.

All three are recorded on Gemessenes-Verhalten in workflow-util/pipelines, which is where measurements about the forge itself belong; they are repeated here only because the run happened on this side.

Answered since, by run 399: the analysis completes (44 s, ANALYSIS SUCCESSFUL). What it found is still unseen from here, and for a structural reason rather than a missing run — the server refuses anonymous API reads (401 on api/measures/component), so the findings are only visible to an account on it. Whether exclusions are warranted therefore depends on a look at the dashboard, not on another run; if they are, they go into sonar-project.properties.

workflow_dispatch is on the hull for exactly these retries, and it is what proves the corrected key: one dispatch, no dummy commit on main. Only the maintainer can trigger it — the bot account cannot dispatch workflows.

ci/helm-dependencies.sh — the step they all share

Not a workflow but the prerequisite of every helm command in this repo: it fetches a chart's declared dependencies into charts/, and returns silently for the charts that declare none (all but one).

It runs in eight places, because every helm command exits 1 while a declared dependency is missing: before lint and package in validate-charts.yml, before package in the shared publish block, and inside ci/run-install-test.sh, ci/run-quantity-render-test.sh, ci/run-selector-match-test.sh, ci/run-httproute-match-test.sh, ci/run-yaml-scalar-render-test.sh and ci/run-trivy-config.sh. The second of those is now across a repository boundary: the block calls this script from the calling repository's checkout, which makes the script part of the contract — renaming or deleting it breaks publishing from a place that no longer reads our diffs. Four of those call sites are the render tests that cover every chart on every PR — so a dependency-carrying chart can turn a third-party Helm repository's bad day into a red pipeline on an unrelated PR, now from either of them. Measured while writing the selector-match test: the registry answered 503 for several minutes and both tests went red on stalwart, ytdlp-web-player and paperless-ngx with nothing in the diff to blame.

It runs helm dependency update, not build: Chart.lock is not committed, because Renovate rewrites Chart.yaml and cannot regenerate the lock, so a committed one would arrive stale.

A PR that touches only ci/

Both matrices follow the changed chart directories, so a PR that only changes the harness exercises nothing by itself. The Trivy scan has a fixed-sample fallback for that case; the install-test has none — bump the affected chart's patch version to pull it into the matrix, or run the all-charts install-test once the change is on main. The quantity render test, the selector-match test, the HTTPRoute render test and the YAML-scalar render test are the four exceptions to the rule: all of them always run over everything. The bump-level job is a fourth exception of a different kind — a PR that changes only ci/renovate-bump-chart.sh is exercised by it, since that job tests the harness rather than the charts.

What a Gitea runner does differently

All measured on this instance, not taken from documentation, and the reason for every deviation in validate-charts.yml.

Reusable workflows with secrets: inherit work, so the 25 thin publish hulls run unchanged — including now that the workflow they call lives in another repository. One quirk: a missing secret does not fail the job, it fails job creation — the run then shows no steps at all.

Runner labels mean different machines. ubuntu-latest has no Docker socket here; ubuntu-docker has one. Exactly one job cares — the install-test, because k3d is a Docker client — so its runs-on is an expression rather than a fixed label, and the same file still works on a GitHub runner where ubuntu-docker does not exist.

A matrix runs concurrently inside ONE job, under ONE status context. Operationally: all legs share one Docker daemon, which makes a per-leg resource name mandatory. For branch protection: a required check per chart is not possible, and the context name carries the unexpanded matrix expression, so required-check names must be read off a real run — the more so since the install-test job moved into a called workflow, which may add the calling job's name to the context. What main does require is validate-charts, as that single context, alongside a review — the two conditions Renovate's auto-merge waits for (see Renovate → PR policy).

Expected runtimes, so a later deviation is visible as one:

measured here GitHub, same commit
Create k3d cluster 19–25 s 22–23 s
install-test, two charts in the matrix (cyberchef + freesailarr) 210–216 s over three runs, spread 6 s ~145 s wall, two parallel VMs
install-test, freesailarr alone, ten containers and a real tunnel 225 s not comparable
install-test, 20 charts in the matrix — the accidental all-charts run of Gitea PR #48 12m41s, one job, all legs concurrent; 19 of 20 Ready never ran there

The runner is not slower — cluster creation is a touch faster. The difference in total time is structural: two matrix legs share one runner here, while GitHub gives each leg its own VM. Reading it as "Gitea is slower" leads to the wrong conclusion.

aquasecurity/setup-trivy cannot be used. It runs actions/checkout against aquasecurity/trivy on github.com and hands it github.token — which here is a Gitea token. github.com answers 401 and checkout ends on fatal: could not read Username for 'https://github.com', naming neither the action nor the token. Its actions/cache step is inert here as well, and fetching the nested actions cost 4m30s before the job even started. The workflow downloads the release tarball instead: one request, same behaviour on either forge, and the # renovate: comment still keeps TRIVY_VERSION current.

Tools are installed, not inherited — and the two cases differ on purpose. kubectl is pinned to match K3S_IMAGE and guarded against version skew (more than one minor apart fails the job with both versions named), because a client's distance to the API server changes what the test means. envsubst is only checked for and fetched if absent, because it substitutes text and the release doing so does not change the result; pinning it would be upkeep without value. That GitHub's image ships both does not make either unnecessary — it only makes the drift invisible there.

Containerisation is detected, not inferred from the platform. k3d publishes the API server port on the Docker host; when the job is itself a container — as every Gitea Actions job is — that host is the dind container, so the https://0.0.0.0:<port> k3d writes into the kubeconfig points at the job container, where nothing listens. The workflow asks the daemon about its own container instead of asking which forge it is on: a self-hosted GitHub runner in a container has the identical problem, and a condition that says "GitHub" while meaning "not containerised" is the kind of lie this repo exists to remove.

Still unmeasured, because they never stood in the way: /dev/fuse and user namespaces. Said as unmeasured rather than as fine.

What a GitHub runner had been providing quietly

Getting the install-test to run on Gitea surfaced six assumptions nobody had written down. Five are about the runner and fail loudly, fixed on the node side: Docker bridges built at MTU 1500 against a pod network of 1450 (small responses fit in one packet, a 17 MB tarball does not, so it looked like a blocked host); fs.inotify limits too low for k3s, so containerd never started and the node never registered; no kubectl and no envsubst in the image; and the kubeconfig pointing at 0.0.0.0.

The sixth is about our code, and it is the one to remember — see Lessons learned: the k3d cluster name was hard-coded, which was wrong on GitHub too and simply could never show there.

Two rules for every check added here

  1. A check that a broken system passes is worse than no check. kubectl get nodes exits 0 on an empty list, so a readiness check written that way went green against a cluster whose node had never registered — and the failure surfaced five minutes later, somewhere unrelated. kubectl wait --for=condition=Ready nodes --all is not the shorter way to write it: with zero nodes the condition is vacuously satisfied. Assert the thing you mean — at least one node, and Ready.
  2. Green does not mean checked. A passing job earns exactly as much trust as the assertions inside it. The open case of this kind: the images in ci/<chart>/fixtures.yaml are not followed by Renovate, so the install-test can keep passing against fixture infrastructure that is years stale (Gitea #24). Not solved — named so it is not mistaken for solved.

change-app-version.yml

A manually dispatched app-version bump that rewrites version:/appVersion: and commits directly to main, bypassing both the validate and the install-test pipelines. It still exists for one-off bumps, but the normal path for an app update is a Renovate PR. A new chart must be added to its chart-name choice list.

helm-charts wiki

How we work

How it is built

How charts are written

Charts